November 2023 – Learning Aloud

Turning AI on my content

In reviewing the various ways I might use AI, I am starting to see a pattern. There are uses others are excited about that are not relevant to my life. There are possible uses that are relevant, but I prefer to continue doing these things myself because I either enjoy the activity or feel there is some personal benefit beyond the completion of a given project. Finally, there are some tasks for which AI serves a role that augments my capabilities and improves the quality or quantity of projects I am working on.

At this time, the most beneficial way I use AI is to engage an AI tool in discussing a body of content I have curated or created as notes and highlights in service of a writing project I have taken on. There are two capabilities here that are important. First, I value the language skills of an AI service, but I want the service to use this capability only as a way to communicate with me about the content I designate. I am not certain I know exactly what this means as it would be similar to saying to an expert with whom I was interacting tell me about these specific sources without adding in ideas from sources I have not asked you to explore. Use your general background, but use this background only as a way to explain what these specific sources are proposing. What I mean is don’t add in stuff to address my prompt that does not exist within the sources I gave you.

Second, if I ask an AI service about the content I have provided, I want the service to be able to identify the source and possibly the specific material within a source that was the basis for a given position taken. Think of this expectation as similar to the expectation one might have in reading a scientific article to which the author provides citations for specific claims made. My desire here is to be able to evaluate such claims myself. I have a concern in simply basing a claim on the language of sources not knowing the methodology responsible for producing data used as a basis for a claim. For serious work, you need to read more than the abstract. Requiring a precise methodology section in research papers is important because the methodology establishes the context responsible for the generation of the data and ultimately the conclusions that are reached. Especially in situations in which I disagree with such conclusions, I often wonder if the methodology applied may explain the differences between my expectations and the conclusions reached by the author. Human behavior is complex and variables that influence behavior are hardly ever completely accounted for in research. Researchers do not really lie with statistics, but they can mislead by broad conclusions they share based on a less-than-perfect research method. There are no perfect research methods hence the constant suggestion that more research is needed.

Several services approximate the characteristics I am looking for. I will identify three such services. I had hoped to add a fourth, but I intended to subscribe to the new OpenAI applications recently announced, but the $20 a month subscription fee necessary to use these functions was recently suspended so I will have to wait to explore these functions until OpenAI decides to expand the user base.

The three services I have worked with include NotebookLM, Mem.ai, and Smart Connections with Obisidan. I have written about Mem.ai and Smart Connections in previous posts, so I will use NotebookLM for extended comments and then offer quick examples produced by the other two services.

NotebookLM

NotebookLM recently was made available to me so I have had less experience using this tool than the other two I have included.

At this time you can create a notebook based on 5 pdfs or 10 text segments. There is a limit to how much content these units of information can total, but the pdfs I used were research articles so this is a lot of content. In the following image, icons for the 5 pdfs appear in the column on the left of the screen. All of the pdfs have been selected for the interaction I intend with the AI tool and a check beside each pdf indicates this is the case. The window for submitting my prompts appears at the bottom of the image.

The second image shows the results from the prompt I have submitted (I will explain my interest in interacting with these pdfs following this description). The response to this query is provided above the prompt window (I will share the text) and below this window you will see that the summary was based on 10 “citations”. Selecting one of these citations brings up the content that now appears in the left-hand column. This window shows the content of the pdf and some summary comments about that document generated by the AI. The AI also proposes additional prompts a user might want to apply to that content.

The task I asked all three services to address was related to a recent post I wrote describing generative cognitive tasks and comparisons among the impacts of these generative activities. The pdfs were research studies I had reviewed and were the basis for that post. I had detailed familiarity with this content from both reviewing and annotating this material and writing the previous post and I use this approach of first engaging a process without AI to provide a basis for commenting on what the AI provides. The prompt that follows is what I submitted to NotebookLM to see what would be generated. I will use a similar prompt with the other two AI services/

Prompt: Use only my notes, generate a 400 word summary of generative learning activities and include what is known about the relative effectiveness of different activities.

NoteBook response

Generative learning is a type of learning in which students are actively engaged in creating new knowledge by interacting with the material. This is in contrast to passive learning, in which students simply listen to a lecture or read a textbook.

Generative learning activities are designed to encourage students to think critically about the material and to make connections between new information and their existing knowledge.

There is a significant body of research evidence that supports the effectiveness of generative learning activities. A meta-analysis of 500 studies found that generative learning activities resulted in better learning outcomes than traditional, passive learning methods (Coffey et al., 2003).

Some of the most effective generative learning activities include:

Summarizing: Students are asked to summarize a text in their own words. This activity helps students to identify the main ideas of the text and to understand the relationships between them.
Questioning: Students are asked to generate questions about a text. This activity helps students to think critically about the text and to engage with the material in a deeper way.
Discussion: Students are asked to discuss a text with their classmates. This activity helps students to share their understanding of the text and to learn from each other.

The relative effectiveness of different generative learning activities depends on the specific learning goals. For example, summarizing is a good activity for helping students to remember information, while questioning is a good activity for helping students to understand the material in a deeper way.

Generative learning activities can be used in a variety of settings, including K-12 classrooms, higher education, and corporate training. They are a valuable tool for helping students to learn and remember new information in a more effective way.

***

You will note in this material that NoteBook included a reference. This was not one of the pdfs I provided for review so I thought it might have been included in the reference section of one of the pdfs. I asked Notebook to provide a full citation and it generated the following.

Coffey, J. E., & McDonald, J. (2003). The effects of generative learning activities on student achievement: A meta-analysis. Contemporary Educational Psychology, 28(4), 240-263. doi:10.1016/j.cedpsych.2003.09.001

While this citation looks authentic and has the proper APA format, it is made up. I tried searching Google Scholar and used my library online access to search the location in Contemporary Educational Psychology and no such article exists. Google asks for user feedback and I provided the fake citation in my evaluation.

***

Mem.AI

My use of Mem.ai and Smart Connections require a little different approach. The content stored in these services results from my exporting notes and highlights from the research articles I read. Both services are directed at my entire collection of notes so this is different from what NotebookLM could access. However, the citations indicated that the output was based on very similar source material.

Prompt: Use only my notes, generate a 400 word summary of generative learning activities and include what is known about the relative effectiveness of different activities.

Mem.ai response

Generative learning is a process where learners actively engage with material, creating new knowledge and connections. This approach involves strategies such as summarization, teaching, self-testing, and self-explaining, which require learners to select and organize relevant information, elaborate on the material based on personal knowledge, and integrate new information with existing knowledge ^{1^}.

Fiorella and Mayer (2016) identified eight strategies that promote generative learning, including summarization, mapping, drawing, imagining, self-testing, self-explaining, teaching, and enacting. These strategies help learners to create a product based on existing knowledge ^{1^}.

Brod (2021) focused on age differences in generative learning strategies and emphasized the importance of producing a meaningful product that goes beyond simple paraphrasing or retelling. For instance, highlighting does not result in new content, but summarization does. Brod’s review suggests that summarizing becomes useful from high school onwards, while testing is beneficial for all ages ^{2^}.

Chi’s (2009) framework differentiates learning activities into active, constructive, and interactive categories. Active engagement with the material can activate existing knowledge and store new input, while constructive activities involve creating a product based on existing knowledge. Interactive activities involve engaging with others to produce a product that integrates external input with personal knowledge ^{1^}.

Fiorella (2023) proposed a new framework of sense-making that distinguishes among three sense-making modes—explaining, visualizing, and enacting—that serve unique and complementary cognitive functions. The explaining mode generalizes knowledge, the visualizing mode organizes knowledge, and the enacting mode simulates knowledge ^{3^}.

In conclusion, generative learning involves actively engaging with the material to create new connections and deeper understanding. The effectiveness of different generative learning activities can vary based on factors such as age and the specific learning context. However, strategies such as summarization, teaching, self-testing, and notetaking with revision and elaboration can promote generative learning and enhance understanding of the material.

Summarize my notes on generative learning Tue 4/18

Brod’s generative Fri 4/14

Fiorellaet2023 Fri 10/13

1^: Summarize my notes on generative learning

2^: Brod’s generative

3^: [Fiorellaet2023](https://mem.ai/m/cqo3CLWb5tkb8syut

At the end of each paragraph in this output, you see a number. The number corresponds to the citation at the bottom. About the three links that take a user to that note, you see the names of the notes. So, this output is carefully linked to sources. I will provide an example of what my notes look like in association with the output from Smart Connections, but each note contains a citation of the article from which I created that note.

***

Smart Connections (within Obsidian)

Obsidian contains my notes from research articles I have read and also highlights from other sources provided to Obsidian through Readwise. I tell you this again because I do not have a comparison among AI services based on exactly the same base of resources. However, again, you will note similar sources are found. Note also that the prompt is a little different to get Smart Connections to include citations.

Prompt: Use only my notes, generate a 400 word summary of generative learning activities and include what is known about the relative effectiveness of different activities. Include links to the individual source notes within your summary

Response:

Based on your notes, generative learning involves actively making sense of new information by mentally reorganizing and integrating it with one’s prior knowledge (Fiorella and Mayer, 2016). It is a constructive process that requires learners to select important information, reorganize it, and relate it to what they already know (Fiorellaetal2016).

Fiorella and Mayer (2016) identified eight strategies that promote generative learning: summarizing, mapping, drawing, imagining, self-testing, self-explaining, teaching, and enacting. Summarizing involves selecting the main ideas, organizing them into a logical structure, and integrating new information with prior knowledge (Fiorellaetal2016). It is more effective for relatively simple concepts and requires training to use effectively (Fiorellaetal2016).

Mapping and drawing also involve transforming information into different forms of representation (Fiorellaetal2016). These strategies help learners build connections and organize the material (Fiorellaetal2016).

Self-testing, or retrieval-based learning, requires learners to actively retrieve information from memory without access to the provided materials (Fiorellaetal2016). It strengthens memory and makes it more accessible in the future (EndresHigh).

Self-explaining involves explaining the content of a lesson to oneself during learning (Fiorellaetal2016). It promotes meaningful learning outcomes when learners engage in reflective knowledge building (Fiorellaetal2016).

Teaching, on the other hand, involves selecting relevant information, organizing it into a coherent structure, and elaborating on it by incorporating existing knowledge (Fiorellaetal2016). It sets itself apart from other generative strategies by focusing on helping others learn (Fiorellaetal2016).

Based on the research by Fiorella and Mayer (2016), it is more appropriate to consider each generative strategy as suitable for certain types of learning and learning contexts rather than regarding one strategy as the most effective (Fiorellaetal2016).

In a study by Koh, Lee, and Lim (2018), the learning benefits of teaching were investigated. The results showed that teaching improved learning outcomes when it involved retrieval practice, but not when learners used a prepared script (Generative > TeachingAsRetrieval). This supports the idea that retrieval practice is an effective generative learning activity.

Overall, generative learning activities have been shown to be effective in promoting meaningful learning outcomes. However, the effectiveness of each activity may vary depending on the complexity of the material and the training provided to learners. It is important to consider the specific learning goals and contexts when selecting and implementing generative learning strategies (Fiorellaetal2016).

Based on your notes, here are the individual notes that provide more information on each generative learning activity:

– [[Generate1]]: Provides an overview of generative learning theory and the eight strategies identified by Fiorella and Mayer (2016).

– [[Summarizing]]: Describes the summarizing strategy and its effectiveness in selecting main ideas and creating quality summaries.

– [[Teaching]]: Discusses the goal of teaching and its distinction from other generative strategies.

– [[Self-Testing]]: Explains the self-testing strategy and its role in retrieval-based learning.

– [[Promote generative learning]]: Provides a link to the article by Fiorella and Mayer (2016) that identifies the eight strategies and discusses their effectiveness.

– [[Fiorellaetal2016]]: Summarizes the key points from Fiorella and Mayer’s (2016) article on generative learning strategies.

– [[EndresHigh]]: Discusses the importance of task differences in retrieval practice and the effects of different types of recall tasks.

– [[Yang2021]]: Highlights a meta-analysis on the effectiveness of retrieval practice in classroom learning and raises questions about the mechanism and student perception of its benefits.

***

Note: The output of Smart Connections is in markdown and the terms included in double parentheses are links that connect to the source note. So, if the summary above was opened in an app that would interpret markdown, the text within the double parentheses would appear as a link and the link would take me to a file stored on my computer. The file is named Generate1.

Here is an example of one of the original notes that was identified as source material.

Generative learning makes sense of new information by reorganizing it and relating it to existing knowledge. This position comes from Wittrock, but is similar to other theorists (Mayer, Piaget). This specific article identified eight learning strategies that promote generative learning and provides a review of research relevant to each strategy.

[[Summarizing]]

Mapping

Drawing

Imagining

[[Self-Testing]]

Self-Explaining

[[Teaching]]

Enacting

The first four strategies (summarizing, mapping, drawing, and imagining) involve changing the input into a different form of representation.

The final four strategies (self-testing, self-explaining, teaching, and answering practice questions) require additional elaboration.

Fiorella, L., & Mayer, R. E. (2016). Eight ways to promote generative learning. _Educational Psychology Review, 28(4), 717-741.

***

Summary

Keeping in mind my recognition that the AI of the three AI services was applied to slightly different content, I would argue that Smart Connections and Mem.ai are presently more advanced than NotebookLM. Eventually, I assume a user will be able to direct NotebookLM at a folder of files so the volume of content would be identical. Google does acknowledge that Notebook is still in the early stages and access is limited to a limited number of individuals willing to test and provide feedback. The content generated by all of the services was reasonable, but NoteBook did hallucinate a reference.

My experience in comparing services indicates it is worth trying several in the completion of a given task. I have found it productive to keep both Smart Connections and Mem.ai around as the one I find most useful seems to vary. I do pay to use both services.

Cornell Notes and Beyond

While a research assistant at Cornell, Walter Pauk was credited with the development of the Cornell Note-taking system. Cornell notes became widely known through Pauk’s popular book “How to study in college” first published in 1962 and available through multiple editions. I checked and Amazon still carries the text.

Pauk’s approach which can be applied within a traditional notebook involves dividing a page into two columns with the right-hand column about twice as wide as the left-hand column and leaving a space across the bottom of each page for writing a summary. The idea is to take notes during a presentation in the right-hand column and later follow-up in the left-hand (often called cue column) with questions and other related comments. This second pass is supposed to follow soon after class so that other memories of the presentation are still fresh. The summary section provides a space to add just what it says – a summary of the main ideas.

Paul explained the proper way to use his system as the five Rs of note-taking. In my experience, the 5 Rs are far less well-known and yet important because they explain how the basic system is to be used. I would organize and explain the 5 Rs as follows.

During class – Record

After class – Reduce

Over time

Recite (cover notes and see what you can recall based on cues)

Reflect (add your own ideas, extensions)

Review (review all notes each week)

While the Cornell system was designed during a different time and was suited to the technology of the day (paper and pencil), those who promote digital note-taking tools offer suggestions for applying the Cornell structure within the digital environment of the tool they promote.

Cornell notes within Obsidian

Cornell notes within Notion

When I used to lecture about study skills and study behavior, I explained the Cornell system, but I would preface my presentation with the following questions. How many of you have heard of Cornell Notes? The SQ3R system? More had heard of Cornett notes and a few of SQ3R. I would then ask are any of you now using either of these systems to study my presentations or your textbook. In the thousands of students I asked, I don’t remember anyone ever raising her or his hand. To test my approach, I also asked if any student made and study note cards in their classes. The positive responses here were much more frequent. I tried to get a sense of why without much luck. I think my data are accurate and I raise this experience to get you to consider this same question. Students take notes, but don’t have a system.

I think Cornell notes are frequently proposed and taught to younger learners because the design of the note collection environment is simple and easy to describe. I wonder about how the process is communicated and perhaps more importantly implemented. The structure makes less sense if students are only intending to cram rather than frequently review. Does the learner have to “buy in” to the logic or do learners understand the logic, but just are not motivated to put in the effort? How any method is taught and understood likely has at least some impact on whether suggestions are implemented.

Understanding Cornell Notes at a deeper level

Note-taking has always been a personal interest and my posts have frequently commented on note-taking. I may have mentioned Cornell notes in a few of these posts, but my focus tends to be on a more basic level. If I am describing a system, what about specific components of that system have known cognitive benefits to learners?

I come to the interpretations of those advocating specific study strategies from a cognitive perspective trying to analyze those strategies from this perspective. I ask what about a given study strategy seems like it makes sense given what those who study human cognition have found that benefits learning, retention, and transfer (application). What in a given study strategy could be augmented or given additional emphasis based on principles proposed by cognitive researchers? I will now try to apply this strategy to Cornell notes. I don’t know enough about Pauk’s work to know his theoretical perspective when creating this approach. For the most part, the perspective I take in my analysis has followed Pauk’s work which occurred during the 1950s. Timelines in this regard do not require that research precede practice, but there is a possibility that new research may offer new suggestions,

Topics

My comments will be organized as three topics.

Stages of study behavior – how should the activities intended to benefit learning occur over time. What should be done when?

Generative experiences and a hierarchy of such experiences – My explanation of a generative activity is an external activity intended to encourage a productive cognitive behavior. By hierarchy, I am pointing to research that has attempted to identify more and less effective generative activities and explain what factors are responsible for this ranking.

Retrieval practice / testing effect – Research demonstrates that activities requiring the recall of stored information increases the probably of future recall and also increases understanding. Testing – free recall, cued recall, and recognition tasks – are common, but not the only or necessarily the most effective ways to engage retrieval effort.

Stages of study behavior

My personal interest in note-taking can be traced to the insights of Di Vesta and Gray. These researchers actually differentiated functions – encoding and external storage, but these processes were really centered within the stages of taking notes and then review. Encoding interpreted more broadly can occur at multiple points in time and this is my point in recognizing stages.

Pauk clearly recognized stages of study in proposing that learners function according to the 5Rs. The original notes were to be interpreted, augmented, and reviewed several times between the original recording and the immediate preparation for use.

Luo and colleagues proposed that notetaking should be imagined as a three-stage process with a revision or update stage recognized after notetaking and before final preparation for use. In addition to recognizing the importance of following up to improve the original record, these researchers advocated for collaboration with a partner. Students do not take complete notes and the opportunity to compare notes taken with others allows for improvements. Research included in the paper points to the percentage of important ideas missed in the notes most record. The authors propose that lectures pause during presentations to provide an opportunity for comparison.

This source describes studies with college students using this pause and update method. Students were given two colored pens so additions could be identified. The pause and improve condition generated a significant achievement advantage (second study). However, this study found no benefit when comparing taking notes with a partner vs alone. Researchers looked at notes added and found few elaborations.

In an even more recent focus on multiple stages as part of a model for building a second brain, Forte described a process called distillation or progressive summarization. In this process focused on taking notes from written sources, original content is read using an app that allows the exportation of the highlighted material. This content is first bolded and then highlighted to identify key information (progressive distillation). A summary can then be added. The unique advantage in this approach is to keep all of the layers available. One can function at different levels from the same immediate source and backtrack to a more complete level should it become necessary to recall a broader context or to take what was originally created in a different direction.

It is possible to draw parallels here between what the Cornell system allows and what Forte proposes. The capability of reinstating context and addressing information missing from the original notes is also an advantage of the digital recording of an audio input keyed to specific notes as they are taken (see SoundNote).

Di Vesta, F. & Gray, S. G. (1972). Listening and note taking. _Journal of Educational Psychology, 63_(1), 8-14.

Forte, T. (2022). Building a second brain: A proven method to organize your digital life and unlock your creative potential. Atria Books.

Luo, L., Kiewra, K. A., & Samuelson, L. (2016). Revising lecture notes: How revision, pauses, and partners affect note taking and achievement. Instructional Science, 44(1), 45-67.

Hierarchy of generative tasks

Again, a generative experience is an external activity intended to encourage productive activities. These productive activities may occur without any external tasks and this would be best situation because there is overhead in implementing the external tasks. However, for many learners and for most under some situations, the external tasks require cognitive activities that may be avoided or remain unrecognized as a function of poor metacognition or lack of motivation.

Many tasks initiated by a learner or educator can function as a generative function. Fiorella and Mayer (2016) have identified a list of eight general categories most educators can probably turn into specific tasks. These categories include:

Summarizing
Mapping
Drawing
Imagining
Self-Testing
Self-Explaining
Teaching
Enacting

Immediately, summarization can be identified from this list as being included in the Cornell system. Self-testing would also be involved in the way Pauk described recitation.

What I mean by a hierarchy as applied to generative activities is that some activities are typically more effective than others.

Chi offers a framework – active-constructive-interactive – to differentiate learning activities in terms of observable overt activities and underlying learning processes. Each stage in the framework assumes the integration of the earlier stage and is assumed more productive than the earlier stage.

Active – doing something physical that can be observed. Highlighting would be another example.

Constructive – creating a **product** that extends the input based on **what is already known**. For example, summarization.

Interactive – involves interaction with another person – expert/learner, peers – to produce a product.

One insight from this scheme is that there is a stage beyond what might seem to be the upper limit of the Cornell structure (i.e., summarization). I am tempted to describe this additional level as application or perhaps elaboration. Both terms to me imply using information.

Chi, M. T. (2009). Active?constructive?interactive: A conceptual framework for differentiating learning activities. Topics in cognitive science, 1(1), 73-105.

Fiorella, L., & Mayer, R. (2016). Eight Ways to Promote Generative Learning. Educational Psychology Review, 28(4), 717-741.

Retrieval Practice

Retrieval practice is a learning technique that involves trying to recall information from memory (see also Roediger & Karpicke). There are several reasons why retrieval practice improves future retrieval, but also understanding. First, it forces learners to actively engage with the material. This helps to create stronger connections between the information and existing knowledge. I think of retrieval as looking externally into memory to try to find something connected to what I am searching to find. This makes sense if you understand memory as a web of connections among ideas. The efforts to find specific information results in the activation and awareness of other information in order to find a connection to what is desired.Exploring retrieval not only increases the strength of connection to the desired information, but also an exploration of potentially related information resulting in new insights.

Second, retrieval practice provides feedback on what has been learned and what needs more attention. This helps learners to identify areas where they need to improve.

Retrieval practice is sometimes called the testing effect and asking questions or being asked questions is one way to trigger the search process (e.g., Yang and colleagues), Self testing is an activity embedded in the way Pauk imagines the use of Cornell notes. I am guessing it is also a reason the strategy of making and using flash cards is such a common study strategy.

There are however other ways to practice retrieval. Yang and colleagues speculate that retrieval practice plays in role in the proven benefits of a learner teaching and preparing to teach. Teaching represents an important link here to the more productive levels of generative learning (see previous section). The previously mentioned hierarchy attributed to Luo and colleagues recognized the value of collaboration in reviewing notes and again the addition of sharing and discussion would represent important extensions of a personal use of any note-taking system.

Koh, A. W. L., Lee, S. C., & Lim, S. W. H. (2018). The learning benefits of teaching: A retrieval practice hypothesis. Applied Cognitive Psychology, 32(3), 401-410.

Luo, L., Kiewra, K. A., & Samuelson, L. (2016). Revising lecture notes: How revision, pauses, and partners affect note taking and achievement. Instructional Science, 44(1), 45-67.

Roediger III, H. L., & Karpicke, J. D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on psychological science, 1(3), 181-210.

Yang, C., Luo, L., Vadillo, M. A., Yu, R., & Shanks, D. R. (2021). Testing (quizzing) boosts classroom learning: A systematic and meta-analytic review. Psychological Bulletin, 147(4), 399-435.

Summary – My effort here was an attempt to cross reference what might be described as a learning system (Cornell Note) with mechanisms that might expain why the system has proven value and possibly allow the recognition of similar components present in other study systems. In addition, I have tried to emphasize that the components of a system may not be understood and applied in practice. Collaboration was suggested as a way to extend the Cornell system.

Trust AI?

I have found that I cannot trust AI for a core role in the type of tasks I do. I am beginning to think about why this is the case because such insights may have value for others. When do I find value in AI and when do I think it would not be prudent to trust AI?

I would describe the goal of my present work or hobby, depending on your perspective, as generating a certain type of blog post. I try to explain certain types of educational practices in ways that might help educators make decisions about their own actions and the actions they encourage in their students. In general, these actions involve learning and the external activities that influence learning. Since I am no longer involved in doing research and collecting data, I attempt to provide these suggestions based on my reading of the professional literature. This literature is messy and nuanced and so are learning situations so there is no end to topics and issues to which this approach can be applied. I do not fear that I or the others who write about instruction and learning will run out of topics.

A simple request of AI to generate a position on an issue I want to write about is typically not enough. Often a general summary of an issue AI generates only tells me what is a common position on that topic. In many cases, I agree with this position and want to generate a post to explain why. I think I understand why this difference exists. AI works by creating a kind of mush out of the content it has been fed. This mush is created from multiple sources differing in quality which makes such generated content useful for those not wanting to write their own account of that topic. I write a lot and sometimes I wish the process was easier. If the only goal was to explain something that was straightforward and not controversial relying on AI might be a reasonable approach or at least a way to generate a draft.

As I said earlier, educational research and probably applied research in many areas is messy. What I mean by that is that studies of what seems to be the same phenomenon do not produce consistent results. I know this common situation leads some in the “hard” science to belittle fields like psychology as not a real science. My response is that chemists don’t have to worry that the chemicals they mix may not feel like responding in a given way on a given day. The actual issue is that so many phenomena I am interested in are impacted by many variables and a given study can only take so many of these variables into account. Those looking to make summary conclusions often rely on meta-analyses to combine the results of many similar studies to achieve a type of conclusion and this approach seems somewhat similar to what AI accomplishes. Finding a general position glosses over specifics.

Meta-analysis does include some mechanisms that go beyond the basic math involved in combining the statistical results of studies. This approach involves the researchers looking for categories of studies within the general list of studies that identify a specific variable and then quantitatively or logically trying to determine if this unique variable modified the overall result in some way.

The approach of examining subcategories is getting closer to what I am trying to do. I think it essential when considering an applied issue to review the methodology of the studies that differ and see what variables have been included or ignored. There is not an easy way to do this. It is not what AI does and it is not something humans can do when simply reviewing the abstracts of research. Do the researchers control a variable you as a consumer/practitioner think may matter? I encounter this issue frequently and I admit this experience often occurs because I have a bias that trends in a different direction than what the data and conclusion of a given study may propose. Biases are always important to recognize, but science relies heavily on doubt and testing alternatives is an important part of the process.

For example, I don’t happen to believe that taking notes by hand and reading from paper are necessarily better than their digital equivalents. I have read most of the studies that seem to make this case, but I find little in the studies that would explain why? Without credible answers to the “why” question, I continue to doubt and since I cannot generate data myself, I continue to examine the data and methodologies of specific studies looking for explanations.

Long intro, but I thought it necessary to support the following point. AI is not particularly helpful to me because conclusions reached from a mess or amalgam of content without specific links to sources I can examine seems a digital representation of the problem I have just outlined. AI searches for a common position when the specifics of situations and may create a common position that is misleading.

If you use AI and haven’t tried this, I would ask that you try working backward. By this, I mean ask the AI system to offer sources that it used in responding to your request. Exploring this approach works best when you know the field and have a good handle on what the AI should produce.

Here is an example I generated recently when trying to understand what factors were responsible for the proposed value of the Cornell note-taking system. Many who read my posts probably recognize this approach to taking notes. What about this system explains why it seems to help students? I was using the AI system Leo which is now available as a tool within the Brave browser.

The following image shows Leo as a window that opens in a Brave side panel (panel is opened using the icon at the top and Leo is activated by the icon enclosed by the red box on the right). At that point, you can being “chatting” with Leo.

Leo gave me a nice analysis of what the Cornell note-taking system is and offered several possibilities for why the system is effective. I then asked Leo for references in support of the reasons offered for effectiveness which resulted in names and dates of studies and I then asked for full citations. Leo responded with the following list and even indicated that it hoped this information would be helpful.

Here are the full citations for the references I provided earlier:

1. Fisher, R. P. (1993). The Cornell note-taking system: A study of its effectiveness in improving lecture note-taking and recall. Journal of Educational Psychology, 85(2), 278-286.
2. Kiewra, K. A. (1994). The Cornell note-taking system: A review of research on its effectiveness. Journal of Learning Disabilities, 27(3), 171-180.
3. Koech, J. (2013). The effect of the Cornell note-taking system on the academic performance of students with learning disabilities. Journal of Learning Disabilities, 46(2), 148-158.
4. Kornell, N., Son, L. K., & Tai, K. (2007). The importance of retrieval for learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33(3), 576-585.
5. Bjork, E. L., & Bjork, R. A. (1992). A new theory of disuse and an old theory of stimulus decay. In A. Healy & R. Brown (Eds.), The psychology of learning and motivation: Vol. 8. Cognition and practice (pp. 115-145). New York: Academic Press.

I hope these references are helpful for you! Let me know if you have any other questions.

Here is the thing about this list. None of these citations is real. The names are appropriate as researchers who work in this area, the titles make sense, and the journals exist. However, try inserting any title from this list in Google Scholar and you will find there are no matches. I tried the more direct approach of actually going to the journals and looking for the papers. There was nothing to find. AI mushes things together and loses track of specifics. Even the names and titles make sense out of this mush, but the origins of the information cannot be retraced and reviewed.

If I were to offer the summary of my request as a blog post, it would be informative and accurate. If I were to append the citations on which this summary was generated, I would find myself embarrassed as soon as someone decided they wanted to use a citation to learn more. Is there value here? I think so as long as a user understands what they are getting. AI seems to do a reasonable job of presenting a summary of what others have written. However, at least within the scenario I have described, it is important to understand limitations. When I challenged Leo on a specific citation, Leo was willing to explain in its own words that it had just made the citation up.

I have come to my own strategy for using AI. I use a tool such as Elicit to identify citations that I read creating my own notes. I then use AI tools to offer analysis or summaries of my content and to identify the notes that were used in generating responses. If it references one of my notes, I am more confident I agree with the associated statement.

This post is already far too long, so here is a link to an earlier post describing my use of Obsidian and AI tools I can focus on my own notes.

To Win the War, Defeat Hamas and Stop Settlements

I write this post in response to comments made during last evening’s Republican debate during which several participants were very negative in response to the discussions on the Israeli/Palestinian situation occurring on many college campuses. The tone advocated that university officials shut down students voicing their concerns that the military actions of the Israelis were indiscriminately killing a large number of Palestinians innocent of any aggression toward Palestinians and voicing recognition of the abuses the Israelis had visited upon those of other faiths in the West Bank and the region. Of specific concern in these vocalizations has been the complicity of leader Netanyahu in using the support of leaders of Israeli extremists in order to gain political power. I support this concern of the students and see little hope for the region unless the overreach of all extremists in the region are not eliminated.

I think students are idealistic and see the hypocrisy in the positions taken by many politicians. Whether these politicians see their position as pragmatic or not is not the issue for me. Pragmatism can easily become a way to ignore legitimate complexity that is acknowledged by others. You do not have to condone atrocities to understand that people placed in impossible life circumstances are easier to convince they have no choice other than to resort to violence. My point is that the students recognize that simplifying a complex situation to justify a given response is unlikely to have long-term success.

In reacting to the simplicity of the Republican candidates in reacting to the present world situation, I happened to read an NYTimes piece by opinion writer Thomas Friedman. Thomas Friedman is one of the few authors who generates content I find so valuable that I purchase any book he writes. This interest goes back many years. I have found his commentary on globalization, climate, and education very insightful. Before these topics were integrated in what I think of his “World is Flat” books, I read his books about what I have always called the Middle East and I believe this area of the world and his experiences covering issues related to this region established him during his early career. One can find posts on this blog going back many years based on my reading of one Friedman book or another.

Friedman’s argument seems to me to encapsulate the position of the students and reflects a more nuanced and I would suggest an informed analysis of the crisis. His position seems captured in his title – To Win the War, Defeat Hamas and Stop Settlements [I think this link should work whether or not you have a digital subscription to the NYTimes]. I think of this argument as suggesting that to solve a problem of this nature you need to assure all participants that you are willing to put yourself in the position of taking the moral high ground. The crisis will not go away until there is a legitimate two-state solution and the extremists in Israel recognize and eliminate their abuse and aggression toward others who have legitimate rights to exist in the region. This is not a matter of determining which side has committed the most egregious acts, this is a matter of suggesting that wrongs exist on both sides and conflict will continue and be defended by one side as long as this is the case.

School and Professional Note-Taking

I have always been interested in note-taking. I explored options for taking notes as a college student and conducted research on note-taking as a professor. Despite the length of time note-taking has been an interest, the combination of the recent reframing of note-taking as Personal Knowledge Management and technology-based tools for making and using notes has brought a critical revelation. There are essential differences between student note-taking and out-of-school note-taking. Maybe that was obvious to everyone else. I don’t think so.

I have come to think about the distinction in this way. Do the tools, methods, and research of note-taking scale? By this I mean do the research, methods, and tools have been dominated by a way of thinking that imagines a learner listening to a lecture in preparation for a test or reading a book in preparation for writing a paper tell us much about taking notes outside of an educational setting. How does this perspective relate to Professional Note-taking?

I may have been aware of Common Place Books and had a file box full of note cards with citations and a few comments about the research articles I had read (later to be converted to digital tools such as EndNote), but I failed to make the connection that these were note-taking techniques and these were techniques that were very different from what students I was lecturing on note-taking and study skills were doing.

It took the book Smart Notes, learning about Nicholas Luhmann and his Zettelkasten, and my exploration of multiple digital note-taking tools (e.g., Obsidian, Mem.ai) for me to begin forming a broader perspective on note-taking. I was still a notetaker, but my activities were very different from what students do. I thought maybe the labels of school and adult notetakers might be an appropriate distinction, but I worked mainly with adults so that was not quite right. I decided maybe the distinction between school and professional notes made more sense. Again, my question – Do the tools, methods, and research of note-taking scale?

School and Professional Note Differences

Delay until use. An obvious difference is the expected time until use of stored information. With school notes, the likely use of recorded information is less than a semester. Maybe grad students would review notes before prelims or comprehensive examinations (the exams students take toward the end of their PhD training), but such exams are becoming more and more uncommon and Professional notes would still be used over a longer period of time. For Professional notes, the actual use of such notes could be delayed by years and perhaps decades.
Control of goals – The reality in student note-taking is that you most likely understand your purpose as preparation for a task that will be assigned by someone else. You might select the topic of a paper you are taking notes to write, but an instructor likely determined the parameters of the paper. Exams present an even more vague and externally defined objective. Not only do you not usually get to select the focus of the exam or whether an exam will be experienced, you are seldom informed specifically what in the notes or assigned readings are a priority for what will be covered by the assessment. Such lack of control matters because the uncertainty probably means more complete notes should be a goal. In a lecture situation, you can work through priorities and likely foci after a lecture, but the speed of presentation may mean that the best “live situation” strategy is to record as much as you can. It is true that the intended use for Professional Notes is often uncertain at the time you record information, but the difference is a matter of degree. Things that have no interest to you can be ignored without anxiety.

The differences I have outlined here are how I see reality. However, I do wonder when one should begin taking Professional Notes. I wish I had some of the notes from my 50 years ago college years. There is also learning that goes on outside of formal education at all ages and it seems recording insights related to such experiences would be of value. I suppose those who keep a diary have some goal like this in mind.

As my perspective on taking notes has become more expansive, I am starting to think that issues such as control and time frame have implications for tools, strategies, and research related to taking notes.

Tools

I have two general comments on the selection of tools. These comments can be differentiated as a) how important is digital technology in taking notes and b) when digital tools are the answer, which tools.

First, should notes be handwritten or recorded digitally using a keyboard. Yes, I know there are ways to transform handwritten notes (on paper or screen) into digital, but I see this approach as on the fringe. My answer here recognizes that the time frame / control of application issues must be part of the answer. For me, the answer is simple – go digital. If you are a fan of some of the research on student use of pen vs. keyboard and are convinced by the data from the studies showing pen and paper are superior – fine. This is not the place for me to argue and explain.

OK, I can’t help myself. If you favor pen/paper, read the methodology of those studies carefully. When was the exam in the research comparison given? More complete notes (what tends to result from keyboards) offer little advantage offer no advantage if exams are immediate and retention over time is less important. What experience do learners have taking notes on a laptop? An important limitation in both notetaking and reading comparisons of paper vs screen is the experience the participants in the studies have with extended use of technology as notetakers or readers.

I think there are obvious advantages of technology that should not be ignored. The comprehensiveness of the notes and the ability to record important information are great predictors of evaluations related to this information. For those who are supporters of the “too much typing limits thinking argument”, I would propose that students use a digital tool that records the audio of the input while the student takes notes. SoundNote makes a good example, but there are several tools of this type. Here is what happens. Notes taken are time-stamped to locations in the audio recording (you don’t see the time stamps). When something in the notes confuses you later, click within your notes and listen to the recording at that point. Very efficient. If you don’t understand, enter something like ???? in your notes and review the audio when you are out of the classroom and can take the time. You cannot do these things with paper.

Summary – for school notes, I would recommend digital notes as soon students can manage. Like any approach to taking notes, there are useful strategies that take advantage of what a tool offers.

Professional Notes and tools – I think Professional Notes benefit from more sophisticated tools and probably the utilization of different tools over time. This combination probably means that the pass-through of content from tool to tool is an issue to consider. A second and related issue is that Professional Notes benefit from some “slicing and dicing” over time. By this, I mean important ideas and concepts can be isolated and then combined and recombined (or at least linked in multiple ways) over time. Part of this process is based on efficient ways to find these ideas.

My personal approach at this time follows. At this time is meaningful in the context of Professional Notes as things change with the advancement of technology and the ability to efficiently migrate content from one tool to a different tool performing similar function or expanded functions is important. I use ReadWise (and its related tool Reader) for ebooks, Notion for web content, and Highlights for pdfs (mostly journal articles) to take notes. I migrate the notes and highlights to Obsidian directly. I also enter individual notes (Smart Notes) directly into Obsidian or code the notes and highlights migrated from the digital note-taking tools with tags and links. I am starting to get into strategies here so I will stop with this description. This sounds complicated, but a) much of this happens automatically or b) can be accomplished by copy and paste.

Strategies

In 1972, DiVesta and Gray proposed what might be described as the classic model of note-taking describing the stages of encoding and external storage. Encoding in this case implied more than a passive transcription of information and proposed generative activities as a benefit of taking notes. I write a lot about generative activities, but here generative might simply be accepted that learners benefit from the cognitive activities in taking notes. From a research perspective, this would be demonstrated by comparing the recall or other measure of understanding of a group taking notes and another group just listening with neither group reviewing notes before attempting the test of understanding.

Kiewra and colleagues recognized that encoding was too vague and proposed that the process of taking notes could result from qualitative or quantitative benefits. More notes would require paying attention more successfully which would be a quantitative benefit. If there was a benefit to taking notes that involved the thinking that was generated in doing so, this would be a qualitative benefit. Einstein and colleagues proposed there was some evidence of a qualitative benefit based on data comparing better and poorer students because better students recalled more of content defined as more important with little difference in the recall of less important information. Some thinking while listening had to be going on to differentiate the attention paid to the more important ideas.

In general, research does show that what happens while taking the original notes is not as important as what follows. This effect is one reason I am in favor of a system that provides a way to capture more rather than fewer notes. With more notes, you have more to work with and what a learner does with this content is where the more significant benefits are produced.

What makes sense to me is to conceptualize note use within a three-stage model. Rather than just understanding taking notes as recording and reviewing effective notetaking might benefit from an intermediate stage – revision. For example, Luo and colleagues investigated revision during pauses in a presentation or for the same amount of time immediately after the presentation and found benefits for more immediate activities working with notes. In thinking about taking notes in terms of activities during three phases that might be described as recording, revision, and review, what was done during revision and review without the time constraints of recording information in real-time during a lecture offers opportunities for cognitive activities that benefit retention and understanding.

Translating this notion of stages into a form that might be familiar to more folks who read my post. Consider the Cornell Note Method (see image below). The tools here are typically paper and pen with pages of paper divided into the three sections shown in the image (Notes, Other, and Summary).

With a blank page divided as I have illustrated, a learner takes notes within the area I have identified as notes. I would describe this as the recording phase. During the revision stage (not a label Pauk would use in explaining his Cornell Note system, but I will use to remain consistent), the learner follows up to react to these notes with questions, insights, issues, etc. that are recorded in the Other section of the page and generates a summary that is added in the section set aside for summary. These additions are generated in reaction to the notes and represent external actions that research has shown encourage useful cognitive activities. These are not activities that would be part of a cramming before an examination study session. Cramming has entirely the wrong connotation for any part of the study process, but I use it here because it is understood and to identify the final phase of preparation before an examination. The process following the initial recording of information assumes a series of returns to the original notes first to post-process the notes to layer on external elements such as questions, comments, and a summary and then to review this combination ideally multiple times.

The Cornell system is a structured tool to some extent because use of the system assumes specific activities will follow the taking of the notes and the tool provides a structure within which these activities record an artifact of these activities.

Tools for Professional Notes – The combination of tools I use in my own process do not structure a sequence of activities and products as concretely as the Cornell system I have described, but I do have a workflow that has some similarities, but also results in different experiences and different products. In response to the greater time delay before application and the uncertainty of how stored information will be applied result in some important differences in activity. The idea of a Second Brain is often used by those focused on Personal Knowledge Management.

A second brain is a system or tool that helps you to collect, organize, and retrieve information. The goal of a second brain is to help you access information more easily, understand the information you have accessed, and apply this information in creative ways. Often, the best approach is to separate ideas from their original context, store important ideas with a newly created context sufficient to make the concept understandable after a significant delay, create a system of metadata that will facilitate rediscovery of these ideas after a significant delay, and offer ways to flexibly connect ideas to creatively produce new insights and products. Tags, links, and powerful search options are important for exploration and rediscovery. Like the revision and review phases I identified in my discussion of School notes, tools for Professional Notes are intended both to be explored on a regular basis (to discover new relationships and revisit ideas looking for new connections) and to use when there is a specific task to complete.

Research

School and Professional Notes have very different research bases. School notetaking and notetaking strategies have been investigated with both laboratory and applied research techniques for years. This research has allowed the development of theoretical models explaining how learning from taking notes happens and what specific strategies seem to work best for which students. As is often the case with applied research in education, the research is probably best described as messy with inconsistent findings possibly related to many different sources of potential variability (learner characteristics, task characteristics, content differences).

The interest in Professional Notes has generated a rich literature on strategies with a very limited literature summarizing studies evaluating effectiveness. I wonder if this will change. The tools and strategies change quickly and data collection would be very difficult because the tools are used in circumstances that would make it difficult to collect data from controlled experiments. In this area, arguments are based more a battle of ideas than a battle of data.

Sources

Di Vesta, F. & Gray, S. G. (1972). Listening and note taking. Journal of Educational Psychology, 63(1), 8-14.

Einstein, G. O., Morris, J., & Smith, S. (1985). Note-taking, individual differences, and memory for lecture information. Journal of Educational Psychology, 77(5), 522-532.

Kiewra, K. A., Mayer, R. E., Christensen, M., Kim, S., & Risch, N. (1991). Effects of repetition on recall and note-taking: Strategies for learning from lectures. Journal of Educational Psychology, 83(1), 120-123.

Luo, L., Kiewra, K. A., & Samuelson, L. (2016). Revising lecture notes: how revision, pauses, and partners affect note taking and achievement. Instructional Science, 44(1), 45-67.