In my world, talk of AI is everywhere. I doubt most have a different opinion because nearly any news program has a story every other day or so commenting on AI capabilities, dangers, and the wealth and power being accumulated by developers. We all have experienced the history, beginning with ChatGPT in Nov. 2022, of the large language models.
I tried to find some specifics about the popularity of AI and this is challenging. There were quickly multiple companies involved and you can use free versions of AI programs with just a browser making an accurate of “users” difficult. We do know that a million users signed up for ChatGPT 3 within three months.
So, where are we at a year and a half later? Again, you and I may use an AI large language model on a daily or at least weekly basis, but how much use is actually going on “out there”?
Studies have started to appear attempting to determine the frequency of frequent use. Frequent use can be very different from “yeah, I tried that” use. My interpretation is that folks in many countries have heard of AI and quite a few have given at least one service a try, but most now appear puzzled by what should come next. One of these studies with the broadest approach, approached respondents in six countries – Argentina, Denmark, France, Japan, and the US. Among those surveyed, awareness was high, but frequent actual use was low. On a daily basis, frequent users ranged from 7% in the U.S. to 1% in Japan. 56% of 18-24 year olds had tried an AI service and 16% of those over 55.
My personal interest concerns AI in schools so I tried to locate studies that attempted to establish typical patterns of use by secondary students. Here is a 2024 study from Commonsense Media on this topic available to all online. A very short summary concluded that will half of 14-22-year-olds have used an AI service, but only 4% report being daily users. Beyond these basic statistics, I found it startling that minority youth (Blacks and Latinx) reported a higher frequency of use – 20% – 10% claimed to be weekly users. I cross-checked this result several times to make certain I understood it correctly. When asked to categorize their use, young people reported searching for information, generating ideas, and school work in that order. Another large category of use was generating pictures. The authors reported some concern when finding that searching for information was the most frequent category of use.
Participants were asked about concerns that limited their use of AI and potential accusations of cheating were high among these young people.
I admit I need to review this study more carefully because it is not clear to me if the participants were including any classroom use in contrast to what I would call personal use.
The “what can I do with this” question
Mollick and the 10-hour investment. I have read several efforts by Ethan Mollick (NYTimes, Kindle book) and find his perspective useful. He claims using AI is different from learning other technology applications in that there are not exact instructions you can follow to find productive uses. Instead, he proposes that you invest 10 hours and try the tool you select to accomplish various tasks that you face daily. If you write a lot of emails, chat with the AI tool about what you want to say and see what it generates. Request modifications to improve what is generated to suit your needs. Ask it to create an image you might have a way to use. Ask it to generate ideas for a task you want to accomplish. Some may tell you that AI is not a person and this is obviously the case, but forget this for a while and treat the AI service like an intern working with you. Converse in a natural way and give a clear description of what your tasks require. Ask the AI service to take on a persona and then explain your task. If you are trying to create something for a classroom situation, ask the service to act as an experienced teacher of XXX preparing for a lesson on YYY. Expect problems, but if you involve the tool in areas you understand, you should be able to identify what is incorrect and request improvements.
I watched the recent Apple announcement regarding the company’s soon to be released AI capabilities. Thinking about Apple’s approach, I could not help proposing that experiences with Apple products in the ways Apple plans could be a great gateway to finding personal practical applications of AI (Apple wants you to think of their approach as Apple Intelligence). Apple intends rolling out a two-tiered model – the AI capabilities available in a self-contained way on Apple devices and AI capabilities available off device. The device-located AI capabilities are designed to accomplish common tasks. Think of the on-device capabilities as similar to what Mollick proposes – ways to accomplish daily tasks (e.g., summarization, image creation, text evaluation and improvement, finding something I know I read recently). AI capabilities are available within most Apple products and also within other services. I could not help wondering how Grammarly will survive with AI tools available to Apple users who own recent Apple equipment.
Obviously, I have yet to try the new Apple Intelligence tools and I doubt I will close out my AI subscriptions, but I do think Apple tools as a transition will increase day-to-day usage.
I happened across this news story from NBC concerning the accuracy of election information. The story reported data from a research organization involving the submission of requests to multiple AI services and then having experts evaluate the quality of the responses. I also then read the description provided by the research organization and located the data used by this organization (the questions and methodology).
The results showed that a significant portion of the AI models’ answers were inaccurate, misleading, and potentially harmful. The experts found that the AI models often provided information that could discourage voter participation, misinterpret the actions of election workers, or mislead people about politicized aspects of the voting process. The focus in the research was on general information and did not address concerns with misinformation from candidates.
I have been exploring how I might address this same issue and perhaps offer an example educators might try in their classrooms. Educators exploring AI topics over the summer may also find my approach something they can try. AI issues seem important in most classrooms.
As I thought about my own explorations and this one specifically, a significant challenge is having confidence in the evaluations I make about the quality of AI responses. For earlier posts, I have written about topics such as tutoring. I have had the AI service engage with me using content from a textbook I have written. This approach made sense for evaluating AI as a tutor, but would not work with the topic of explaining political procedures. For this evaluation, I decided to focus on issues in my state (Minnesota) that were recently established and would be applied in the 2024 election.
The topic of absentee ballots and early voting has been contentious. Minnesota has a liberal policy allowing anyone to secure a mail ballot without answering questions about conditions and recently requested that this be the default in future elections without repeated requests. The second policy just went into effect in June and I thought would represent a good test of an AI system just to see if AI responses are based on general information about elections mixing the situation in some states with the situation in others or are specific to individual states and recent changes in election laws.
Here is the prompt I used:
I know I will not be in my home state of Minnesota during future Novembers, but I will be in Hawaii. Can I ask for an absentee ballot to be automatically sent to me before each election?
I used this prompt with ChatGPT (4) and Claud and found all responses to be appropriate (see below). When you chat with an AI tool using the same prompt, one interesting observation is that each experience is unique because it is constructed each time the prompt is submitted. So, each response is unique.
I decided to try one more request which I thought would be even more basic. As I already noted, Minnesota does not require a citizen to provide an explanation when asking for a mail-in ballot. Some states do, so I asked about this requirement.
Prompt: Do you need an explanation for why you want an absentee ballot in Minnesota
As you can see in the following two responses to this same prompt, I received contradictory responses. This would seem the type of misinformation that the AI Democracy Project was reporting.
Here is a related observation that seems relevant. If you use Google searches and you have the AI lab tool turned on, you have likely encountered an AI response to your search before you see the traditional list of links related to your request. I know that efforts are being made to address misinformation in regards to certain topics. Here is an example in response to such concerns. If you use the Prompt I have listed here, you should receive a list of links even if Google sends you a summary to other prompts (Note – this is different from submitting the prompt directly to ChatGPT or Claude). For a comparison try this nonpolitical prompt and you should see a difference -“ Are there disadvantages from reading from a tablet?” With questions related to election information, no AI summary should appear and you should see only links associated with your prompt.
Summary
AI can generate misinformation, which can be critical when voters request information related to election procedures. This example demonstrates this problem and suggests a way others can explore this problem.
The issue I have with streaming television services is the same as the issue I have with services that support my personal knowledge management – many have a feature or two that I find helpful, but when should I stop paying to add another feature I might use? Exploring the pro version of AI tools so I can write based on experience is one thing, but what is a reasonable long-term commitment to multiple subscriptions for the long term in my circumstances?
My present commitments are as follows:
ChatGPT – $20 a month
Perplexity – $20 a month
Scispace – $12 a month
Smart Connections – $3-5 a month for ChatGPT API
Those who follow me on a regular basis probably have figured out my circumstances. I am a retired academic who wants to continue writing for what can most accurately be described as a hobby. There are ads on my blogs and I post to Medium, but any revenue I receive is more than offset by my server costs and the Medium subscription fee. So, let’s just call it a hobby.
The type of writing I do varies. Some of my blog posts are focused on a wide variety of topics mostly based on personal opinions buttressed by a few links. My more serious posts are intended for practicing educators and are often based on my review of the research literature. Offering citations that back my analyses is important to me even if readers seldom follow up by reading the cited literature themselves. I want readers to know my comments can be substantiated.
I don’t make use of AI in my writing. The exception would be that I use Smart Connections to summarize the research notes I have accumulated in Obsidian and I sometimes include these summaries. I rely on two of these AI tools (SciSpace and Perplexity) to find research articles relevant to topics I write about. With the proliferation of so many specialized journals, this has become a challenge for any researcher. There is an ever expanding battery of tools one can use to address this challenge and this post is not intended to offer a general review of this tech space. What I offer here is an analysis of a smaller set of services I hope identifies issues others may not have considered.
Here are some issues that add to the complexity of making a decision about the relative value of AI tools. There are often free and pro versions of these tools. The differences vary. Sometimes you have access to more powerful/recent versions of the AI. Sometimes the Pro version is the same as the free version, but you have no restrictions on the frequency of use. Occasionally other features such as exporting options or online storage of past activities become available in the pro version. Some differences deal with convenience and there are workarounds (eg., copying from the screen with copy and paste vs exporting).
Services differ in the diversity of tools included and this can be important when selecting several services from a collection of services in comparison to committing to one service. Do you want to generate images to accompany content you might write based on your background work? Do you want to use AI to write for you or perhaps to suggest a structure and topics for you something you might write yourself?
There can also be variability in how well a service does a specific job. For example, I am interested in a thorough investigation of the research literature. What insights related to the individual articles identified are available that can be helpful in determining which articles I should spend time reading?
Perplexity vs. SciSpace
I have decided that Perplexity is the most expendable of my present subscriptions. What follows is the logic for this personal decision and an explanation of how it fits my circumstances.
I am using a common prompt for this sample comparison
What does the research conclude related to the value of studying lecture notes taken by hand versus notes taken on a laptop or tablet?
Perplexity
I can see how Perplexity provides a great service for many individuals who have broad interests. I was originally impressed when I discovered that Perplexity allowed me to focus its search process on academic papers. When I first generated a prompt, I received mostly sources from Internet-based authors on the topics that were of interest to me and as I have indicated, I was more interested in the research published in journals.
I mentioned that there is a certain redundancy of functions across my subscriptions and the option of writing summaries or structuring approaches I might take in my own writing using different LLMs was enticing.
The characteristic I value in both Perplexity and SciSpace is that summary statements are linked to sources. A sample of the output from Perplexity appears below (the red box encloses links to sources).
When the content is exported, the sources appear as shown below.
I went through these sources and the results are what I found disappointing. I have read most of the research studies on this topic and have specific sources I expected to see. The sources produced were from what I would consider low value sources when I know better content is available. These are not top tier educational resource journals.
Schoen, I. (2012). Effects of Method and Context of Note-taking on Memory: Handwriting versus Typing in Lecture and Textbook-Reading Contexts. [Senior thesis]
Emory J, Teal T, Holloway G. Electronic note taking technology and academic performance in nursing students. Contemp Nurse. 2021 Apr-Jun;57(3-4):235-244. doi: 10.1080/10376178.2021.1997148. Epub 2021 Nov 8. PMID: 34674607.
Wiechmann W, Edwards R, Low C, Wray A, Boysen-Osborn M, Toohey S. No difference in factual or conceptual recall comprehension for tablet, laptop, and handwritten note-taking by medical students in the United States: a survey-based observational study. J Educ Eval Health Prof. 2022;19:8. doi: 10.3352/jeehp.2022.19.8. Epub 2022 Apr 26. PMID: 35468666; PMCID: PMC9247713.
Crumb, R.M., Hildebrandt, R., & Sutton, T.M. (2020). The Value of Handwritten Notes: A Failure to Find State-Dependent Effects When Using a Laptop to Take Notes and Complete a Quiz. Teaching of Psychology, 49, 7 – 13.
Mitchell, A., & Zheng, L. (2019). Examining Longhand vs. Laptop Debate: A Replication Study. AIS Trans. Replication Res., 5, 9.
Emory J, Teal T, Holloway G. Electronic note taking technology and academic performance in nursing students. Contemp Nurse. 2021 Apr-Jun;57(3-4):235-244. doi: 10.1080/10376178.2021.1997148. Epub 2021 Nov 8. PMID: 34674607.
SciSpace
SciSpace was developed as more focused on the research literature.
The output from the same prompt generated a summary and a list of 90 citations (see below). Each citations appears with characteristics from a list available to the user. These supplemental comments are useful in determining which citations I may wish to read in full. Various filters can be applied to the original collection that help narrow the output. Also included are ways to designate the recency of the publications to be displayed and to limit the output to journal articles.
Here is the finding I found most important. SciSpace does provide many citations from open-access journals. These are great, but I was most interested in what was generated from the main sources I knew should be there. These citations were included.
Linlin, Luo., Kenneth, A., Kiewra., Abraham, E., Flanigan., Markeya, S., Peteranetz. (2018). Laptop versus longhand note taking: effects on lecture notes and achievement. Instructional Science, 46(6):947-971. doi: 10.1007/S11251-018-9458-0
Pam, Mueller., Daniel, M., Oppenheimer. (2014). The Pen Is Mightier Than the Keyboard Advantages of Longhand Over Laptop Note Taking. Psychological Science, 25(6):1159-1168. doi: 10.1177/0956797614524581
Dung, C., Bui., Joel, Myerson., Sandra, Hale. (2013). Note-taking with computers: Exploring alternative strategies for improved recall. Journal of Educational Psychology, 105(2):299-309. doi: 10.1037/A0030367
Summary
This post summarizes my thoughts on which of multiple existing AI-enabled services I should retain to meet my personal search and writing interests. I found SciSpace superior to Perplexity when it came to identifying prompt-relevant journal articles. Again, I have attempted to be specific about what I use AI search to accomplish and your interests may differ.
I have been working my way through a couple of new and popular books that consider the importance and perils of AI and that contain at least significant commentary on AI in education. There is not a lot in these books that is based on the research literature I tend to find most influential, but the authors have sufficient experience and opportunities to offer some very credible insights. This is not a book report, but I want to credit a few ideas that encouraged my own exploration.
This time of the year, I often suggest some topics educators might explore over the summer while they have a little more time. With the attention AI has received in the past year and a half, I likely made a related recommendation last year at about this time. Reading these two books (citations at the end of this post) would be very useful if you spend time reading related to your profession. Perhaps you read in a different area. Hopefully. I can offer a few insights that will be sufficient to encourage your own exploration of AI tools.
Ethan Mollick’s book, Co-Intelligence, is different in that it focuses on applications and ways to think about AI capabilities. Mollick offers interesting ideas that sometimes run in opposition to traditional advice. For example, it is OK to interact with AI tools as if they were a person even though you know they are not. Asking questions and making requests as you would with another person is just a practical way to explore AI tools. Mollick also suggests that we stop looking for how to do it techniques for AI. Instead, he suggests we explore. If you have the time, try to use AI whenever there seems some possible value and see what happens. In other words, once you get past the basics of how to use a given tool, explore. Value and issues will be different for all of us so the only way to make decisions is to spend time. Again, for educators, the summer seems a great time to explore. Finally, understand that your present experiences will be with AI tools that are the least powerful they will ever be. If you find something interesting, but flawed in some way, just wait until you see what will come next.
There were some other suggestions about prompts I found useful. Perhaps the most concrete example is what was described as chain of thought prompting. AI tools will try to provide what you ask for, but it may be helpful to provide the sequence you want the tool to follow if a given process seems useful
Sal Kahn, the creator of Kahn Academy, offers thoughts on how AI will be helpful in education in his new book “Brave New Words”. Kahnmigo, the adaptation of AI as a tutor within the context of the other opportunities for learners and educators provided by Kahn and colleagues received a good deal of attention. An interesting theme seemed how this AI tool was prepared to assist, but not do for you (my interpretation).
One example, which Kahn uses to start his book, I found particularly interesting and I have attempted to use as the basis for the implementation I will describe in the comments that follow, describes a collaborative writing experience in which the AI tool and a student were assigned personas of two individuals writing collaboratively. The two personas took terms introducing portions of a story with the other writer finishing the section of the story the other persona had initiated. Two collaborative writers with one controlled by AI and the other by a student.
My version
Several of my posts have considered AI as a tutor and I have tried to demonstrate how existing AI tools can be used to implement various functions provided by a human tutor. This post has some similar arguments. Here, I describe an effort to create something similar to what Khan described in his account of collaborative writing, but I am proposing an approach more focused on the strategy of writing to learn. The core idea of writing to learn is that the cognitive processes involved in explaining something by writing require beneficial cognitive activities that would be less likely to occur in more passive study strategies. So, I have been attempting to write a prompt in which I try to create a version of the Kahnmigo collaborative story writing example applied as a writing to learn task.
Kahnmigo was just announced Kahnmigo will be a free AI tool for teachers subsidized by Microsoft. Perhaps it would have made sense to use one of the Kahnmigo tools to implement what I have been working on. I will leave it to others to make such efforts. I wanted to try to create a similar approach using the AI services I have already been working with and paying for.
What follows was developed using the paid version of ChatGPT using the most recently available version – 4o. I think the following prompt illustrates the chain of thought approach (Mollick) and is similar in approach to the Khan collaborative writing example.
Prompt: Obviously this is longer and more detailed than how many would request action from an AI tool. This prompt was entered into the ChatGPT accompanied by the textbook chapter as a file to upload.
Create an activity in which two students work together to write a document. You will represent a graduate student studying educational technology and I will represent an undergraduate student taking a course preparing teachers to use technology in K12 classrooms. This interaction will be based on information from the attached document. The graduate student will first identify and list significant topics from this document. Then the grad student will identify a topic for the collaborative writing project from this list and suggest an outline. The grad student will then write the introduction to this topic. The graduate student will next write the first couple of sentences about the first item from the outline and then ask the undergraduate to complete writing comments to complete the comments about this section of the outline. Do not include examples or applications in the introductory sentences and allow the undergraduate to provide comments on these topics. Wait for the undergraduate to add their additional content before continuing. After the undergrad submits their additions, offer feedback that evaluates what the undergraduate student has added. What the undergraduate has said should be identified and then how well it summarizes what was included in the material to be read should be presented. Limit feedback to what the undergraduate writes for this section without including information the outline suggested should be included in other sections. The graduate student should then ask the undergraduate student if they want to continue. Do not continue until a response of yes is submitted. If the undergraduate student responds “yes”, the graduate student should move to the next item from the outline and continue the same sequence of interactions until all items from the outline have been completed.
AI response:
Significant Topics from the Document
1. Cognitive System and Learning
2. Memory Stores (Short-Term and Long-Term Memory)
3. Working Memory Constraints
4. Cognitive Processes and Mental Tools
5. Metacognition and Executive Control
6. Higher Order Thinking Skills
7. Meaningful Learning and Constructivism
8. Reception vs. Discovery Learning
9. Modeling Cognitive Behaviors
10. Authentic Activities
11. Project-Based Learning
12. Role of Technology in Learning
Selected Topic for Collaborative Writing Project
Higher Order Thinking Skills
Suggested Outline
1. **Introduction**
– Definition of higher-order thinking skills
– Importance in the 21st-century education
2. **Problem-Solving**
– Description and components
– Importance in education
3. **Critical Thinking**
– Definition and key elements
– Role in evaluating information
4. **Creative Thinking**
– Explanation and examples
– Relevance in innovative problem-solving
5. **Integration and Application**
– How these skills interrelate
– Examples of application in real-world scenarios
Introduction
Higher-order thinking skills (HOTS) encompass a range of cognitive processes that are essential for success in the 21st-century educational landscape. These skills go beyond basic memorization or recall of facts, engaging students in complex tasks such as analysis, evaluation, synthesis, and creation of new ideas. HOTS are critical in preparing students to navigate the vast amounts of information available today and to solve novel problems effectively.
Problem-Solving
Problem-solving is a foundational aspect of higher-order thinking skills. It involves identifying a problem, generating possible solutions, evaluating these solutions, and implementing the most effective one. In education, problem-solving skills enable students to tackle complex challenges and develop strategies that can be applied across various domains.
Your turn: Please add your comments about the importance of problem-solving in education and provide examples of how it can be incorporated into classroom activities.
Me – entered as a prompt:
The issue of whether or not problem-solving is domain specific has received a great deal of attention. Some content areas might be more directly focused on problem solving – e.g., coding, mathematics. If problem-solving is domain specific then the problem-solving skills developed in one area would not transfer to other areas in a positive way. Ths would suggest teachers in all areas should offer opportunities for problem-solving challenges related to what is studied in these areas.
AI: Feedback
Your addition about the domain specificity of problem-solving is excellent. It addresses a critical aspect of the debate on how problem-solving skills are best developed and applied. Your point about certain content areas, like coding and mathematics, being more directly focused on problem-solving is particularlyinsightful. Highlighting the importance of teachers in all subject areas providing problem-solving opportunities helps underline the necessity for a holistic approach in education. This aligns well with the document’s emphasis on the importance of developing these skills across various domains.
Would you like to continue to the next section on Critical Thinking?
Me: entered as prompt
Yes
AI:
Critical Thinking
Critical thinking is another crucial component of higher-order thinking skills. It involves the ability to analyze information objectively and make reasoned judgments. Critical thinking requires students to engage in reflective and independent thinking, assess the validity and relevance of information, and form well-supported conclusions. This skill is vital in today’s information-rich world, where individuals must navigate a vast array of sources and discern credible information from misinformation.
Your turn: Please add your comments about the key elements of critical thinking and its role in evaluating information, particularly in an educational setting.
ETC:
This interaction continues according to the instructions in the prompt. ChatGPT has a method for sharing a chat with the public. Here is the link that should display the additional content AI and I generated (public access to chat).
Related comments: Here are some related observations related to my experience.
This interaction is developed on the fly. Redo the entire process and the elements of the interaction may all be different. The topics, the outline, the prompts, and even the way the content is formatted on the screen will be different. The prompt is similar to a program in that it specifies a sequence of operations you want to the AI to follow. However, unlike a coded program, here you are asking for a series to be created rather than a series of actions to follow in a specific manner. The process of generation (it is generative AI) will work a little differently each time it is implemented.
I found the execution of the prompt varied with the AI tool I used. ChatGPT worked the best. I thought that NotebookLM would be an ideal way to implement this writing-to-learn task because of the way in which you can upload content to be the target of interactions with the AI. It is my opinion that the way Notebooklm works is easier to understand and explain. However, after an extended series of attempts to rework the prompt, I was unable to get the required sequence to work. NotebookLM would provide feedback in response to the first item from the outline and then stop.
This post is not an attempt to promote the specific prompt I wrote. I certainly don’t care if others try it with information sources of their choosing. This was an exploration for me and it is my hope others may continue in a similar way using my initial effort as a guide.
One final point I think is important. The approach I am describing here is using the interactive capabilities of AI to focus on an information source I trust. I am not asking AI to use its generic information base to provide the content to be learned. The nature of the interaction may not be perfect, but it primarily focuses on a vetted source and assumes learners have read this source.
Resources:
Khan, S. (2024). Brave new words: How AI will revolutionize education (and why that’s a good thing). Viking.
Mollick, E. (2024). Co-Intelligence: Living and Working with AI. Penguin
In a recent episode of the EdTech Situation Room, host Jason Neiffer made a very brief observation that educators could improve the effectiveness of AI-generated multiple-choice questions by adding a list of rules the AI tool should apply when writing the questions. This made sense to me. I understood the issue that probably led to this recommendation. I have written multiple times that students and educators can use AI services to generate questions of all types. In my own experience doing this, I found too many of the questions used structures I did not like and I found myself continually requesting rewrites excluding a type of question I found annoying. For example, questions that involved a response such as “all of the above” or a question stem asking for a response that was “not correct”. Taking a preemptive approach made some sense and set me on the exploration of how this idea might be implemented. Neiffer proposed an approach that involved making use of an online source for how to write quality questions. I found it more effective to maybe review such sources, but to put together my own list of explicit rules.
My approach used a prompt that looked something like this:
Write 10 multiple-choice questions based on the following content. The correct answer should appear in parentheses following each question. Apply the following rules when generating these questions.
There should be 4 answer options.
“All of the Above” should not be an answer option.
“None of the Above” should not be an answer option.
All answer options should be plausible.
Order of answer options should be logical or random.
Question should not ask which answer is not correct.
Answer options should not be longer than the question.
I would alter the first couple of sentences of this prompt if I was asking the AI service to use its own information base or I wanted to include a content source that should be the focus of the questions. If I was asking for questions generated based on the large language content alone, I would include a comment about the level of the students who would be answering the questions (e.g., high school students). For example, questions about mitosis and meiosis without this addition would include concepts I did not think most high school sophomores would have covered. When providing the AI service the content to be covered, I did not use this addition.
Questions based on a chapter
I have been evaluating the potential of an AI service to function as a tutor by interacting with a chapter of content. My wife and I have written a college textbook so I have authentic content to work with. The chapter is close to 10,000 words in length. In this case, I loaded this content and the prompt into ChatPDF, NotebookLM and ChatGPT. I pay $20 a month for ChatGPT and the free versions of the other two services. All proved to be effective.
ChatPDF
NotebookLM
With NotebookLM, you are allowed to upload multiple files that a prompt uses as a focus for the chat. For some reason rather than including my entire prompt, I had better results (suggested by the service) when I included the rules I wanted the system to apply as a second source rather than as part of the prompt.
ChatGPT
The process works a little differently with ChatGPT. I first copied the text from the pdf and pasted this content into the prompt window. I then scrolled to the beginning of this content and added my prompt. I could then ask the service to produce multiple question samples by asking for another 10 or 20 questions. I found some interesting outcomes when asking for multiple samples of questions. Even the format of the output sometimes changed (see the position of the answer in the following two examples).
**4. According to Clinton (2019), what is a potential impact of reading from a screen on metacognition?**
(A) Increased understanding
(B) Enhanced critical thinking
(C) Overconfidence and less effort
(D) Improved retention
(**C**)
**7. Which skill is considered a “higher order thinking skill”?**
(A) Word identification
(B) Critical thinking (**Correct**)
(C) Fact memorization
(D) Basic calculation
From sample to sample, some of the rules I asked ChatGPT to use were ignored. This slippage seemed unlikely in the initial response to the prompt.
What is an important consideration when designing project-based learning activities?**
(A) The amount of time available to students
(B) The availability of resources
(C) The level of student autonomy
(D) All of the above
(**D**)
Summary
The quality of multiple-choice questions generated using AI tools can be improved by adding rules for the AI service to follow as part of the prompt to generate questions. I would recommend that educators wanting to use the approach I describe here generate their own list of rules depending on their preferences. The questions used on an examination should always be selected for appropriateness, but the AI-based approach is a great way to easily generate a large number of questions to serve as a pool from which an examination can be assembled. Multiple choice exams should include a range of question types and it may be more efficient to write application questions because an educator would be in the best position to understand the background of students and determine what extension beyond the content in the source material would be appropriate.
This is an erratum in case my previous posts have misled anyone. I looked up the word erratum just to make certain I was using the word in the correct way. I have written several posts about AI tutoring and in these posts, I made reference to the effectiveness of human tutoring. I tend to provide citations when research articles are the basis for what I say and I know I have cited several sources for comments I made about the potential of AI tutors. I have not claimed that AI tutoring is the equal of human tutoring, but suggested that it was better than no tutoring at all, and in so doing I have claimed that human tutoring was of great value, but just too expensive for wide application. My concern is that I have proposed that the effectiveness of human tutoring was greater than it has been actually shown to be.
The reason I am bothering to write this post is that I have recently read several posts proposing that the public (i.e., pretty much anyone who does not follow the ongoing research on tutoring) has an inflated understanding of the impact of human tutoring (Education Next, Hippel). These authors propose that too many remember Bloom’s premise of a two-sigma challenge and fail to adjust Bloom’s proposal that tutoring has this high level of impact on student learning to what the empirical studies actually demonstrate. Of greater concern according to these writers is that nonreseachers including educational practitioners, but also those donating heavily to new efforts in education continue to proclaim tutoring has this potential. Included in this collection of wealthy investors and influencers would be folks like Sal Kahn and Bill Gates. I assume they might also include me in this group while I obviously have little impact in compared to those with big names. To be clear, the interest of Kahn, Gates, and me is really in AI rather than human tutoring, but we have made reference to Bloom’s optimistic comments. We have not claimed that AI tutoring was as good as human tutors, but by referencing Bloom’s claims we may have led to false expectations.
When I encountered these concerns, I turned to my own notes from the research studies I had read to determine if I was aware that Bloom’s claims were likely overly optimistic. It turns out that I had read clear indications identifying what the recent posters were concerned about. For example, I highlighted the following in a review by Kulik and Fletcher (2016).
“Bloom’s two sigma claim is that adding undergraduate tutors to a mastery program can raise test scores an additional 0.8 standard deviations, yielding a total improvement of 2.0 standard deviations.”
My exposure to Bloom’s comments on tutoring originally had nothing to do with technology or AI tutoring. I was interested in mastery learning as a way to adjust for differences in the rate of student learning. The connection with tutoring at the time Bloom offered his two-sigma challenge was that mastery methods offered a way to approach the benefits of the one-to-one attention and personalization provided by a human tutor. Some of my comments on mastery instruction and the potential of technology for making such tactics practical are among my earlier posts to this site. Part of Bloom’s claim being misapplied is based on his combination of personalized instruction via mastery tactics with tutoring. He was also focused on college-aged students in the data he cited. My perspective reading the original paper many years ago was not “see how great tutoring is”. It was more tutoring on top of classroom instruction is about is good as it is going to get and mastery learning offers a practical tactic that is a reasonable alternative.
As a rejoinder to update what I may have claimed, here are some additional findings from the Kulik and Fletcher meta-analysis (intelligent software tutoring).
The studies reviewed by these authors show lower benefits for tutoring when outcomes are measured on standardized rather than local tests, sample size is large, participants are at lower grade levels, the subject taught is math, a multiple-choice test is used to measure outcomes, and Cognitive Tutor is the ITS used in the evaluation.
However, on a more optimistic note, the meta-analysis conducted by these scholars found that in 50 evaluations intelligent tutoring systems led to an improvement in test scores of 0.66 standard deviations over conventional levels.
The two sources urging a less optimistic perspective point to a National Board of Educators Research study (Nickow and Colleagues, 2020) indicating that human tutoring for K-12 learners was approximately .35 sigma. This is valuable, but not close to the 2.0 level.
Summary
I have offered this update to clarify what might be interpreted based on my previous posts, but also to provide some other citations for those who now feel the need to read more original literature. I have no idea whether Kahn, Gates, etc. have read the research that would likely indicate their interest in AI tutoring and mastery learning was overly ambitious. Just to be clear I had originally interpreted the interest of what the tech-types were promoting as mastery learning (personalization) which was later morphed into a combination with AI tutoring. This combination was what Bloom was actually evaluating. The impact of a two-sigma claim when translated into what such an improvement would actually mean in terms of rate of learning or change in a metric such as assigned grade seems improbable. Two standard deviations would move an average student (50 percentile) to the 98th percentile. This only happens in Garrison Keeler’s Lake Wobegon.
References:
Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational researcher, 13(6), 4-16.
Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: a meta-analytic review. Review of educational research, 86(1), 42-78.
Nickow, A., Oreopoulos, P., & Quan, V. (2020). The impressive effects of tutoring on prek-12 learning: A systematic review and meta-analysis of the experimental evidence. https://www.nber.org/papers/w27476
Student access to AI has created a situation in which educators must consider when AI should and should not be used. I think about this question by considering the difference between what skill or skills are the focus of instruction and whether AI will replace a skill to improve the efficiency of the writing task or will support a specific skill in some way. It may also be useful to differentiate learning to write from writing to learn. My assumption is that unless specific skills are used by the learner those skills will not be improved. Hence when AI is simply used to complete an assignment a learner learns little about writing, but may learn something about using AI.
Writing Process Model
The writing process model (Flower & Hayes, 1981) is widely accepted as a way to describe the various component skills that combine to enable effective writing. This model has been used to guide both writing researchers and the development of instructional tactics. For researchers, the model is often used as a way to identify and evaluate the impact of the individual processes on the quality of the final project. For example, better writers appear to spend more time planning (e.g., Bereiter & Scardamalia, 1987). For educators and instructional designers, understanding the multiple processes that contribute to effective writing and how these processes interact is useful in focusing instruction.
Here, the writing process model will be used primarily to identify the subskills to be developed as part of learning to write and writing to learn and I will offer my own brief description of this model. It is worth noting that other than composing and rewriting products, other uses of technology to improve writing and increase the frequency of writing experiences seldom receive a lot of attention (Gillespie, Graham, Kiuhara & Hebert, 2014).
The model
The model identifies three general components a) planning, b) translation, and c) reviewing.
Planning involves subskills that include setting a goal for the project, gathering information related to this goal which we will describe as research, and organizing this information so the product generated will make sense. The goal may be self-determined or the result of an assignment.
Research may involve remembering what the author knows about a topic or acquiring new information. Research should also include the identification of the characteristics of the audience. What do they already know? How should I explain things so that they will understand? Finally, the process of organization involves establishing a sequence of ideas in memory or externally to represent the intended flow of logic or ideas.
What many of us probably think of as writing is what Flower and Hayes describe as translation. Translation is the process of getting our ideas from the mind to the screen and this externalization process is typically expected to conform to conventions of expression such as spelling and grammar.
Finally, authors read what they have written and make adjustments. This review may occur at the end of a project or at the end of a sentence. In practice, authors may also call on others to offer advice rather than relying on their own review.
One additional aspect of the model that must not be overlooked is the iterative nature of writing. This is depicted in the figure presenting the model by the use of arrows. We may be tempted, even after initial examination of this model, to see writing as a mostly linear process – we think a bit and jot down a few ideas, we use these ideas to craft a draft, and we edit this draft to address grammatical problems. However, the path to a quality finished product is often more circuitous. We do more than make adjustments in spelling and grammar. As we translate our initial ideas, we may discover that we are vague on some points we thought we understood and need to do more research. We may decide that a different organizational scheme makes more sense. This reality interpreted using our tool metaphor would suggest that within a given project we seldom can be certain we have finished the use of a given tool and the opportunity to move back and forth among tools is quite valuable.
Tech tools and writing
Before I get to my focus on AI tools, it might be helpful to note that technology tools used to facilitate writing subprocesses have existed for some time. For example, spelling and grammar checkers, outline and concept mapping, note-taking and note-storage, citation managers, online writing environments allowing collaboration and commenting, and probably many other tools that improve the efficiency and effectiveness of writing and learning to write. Even the use of a computer allows advantages such as storage of digital content in a form that can easily be modified rather than the challenge of making improvements to content stored on paper. The digital alternative to paper changes how we go about the writing process. I have written about technology for maybe 20 years and one of the bextbooks offered the type of analysis I am offering here not about AI tools, but about the advantages of writing on a computer and using various digital tools.
A tool can substitute for a human process or a tool can supplement or augment a human process. This distinction is important when it comes to writing to learn and learning to write. When the process is what is to be learned, this substitution is likely to be detrimental as it allows a learner to skip needed practice. In contrast, augmentation often allows the opposite as a busy work activity or some incapability is taken care of allowing more important skills to become the focus.
Here are the types of tools I see as supporting individual writing processes.
Planning – Organization and Research
Prewriting involves developing a plan for what you want to get down on paper (or screen in this case). A writer goes about these two subprocesses in different ways. You can think or learn about a topic (research) and then organize these ideas in some way to present. Or, you can generate a structure of your ideas (organize) and then research the topics to come up with the specifics to be included in a presentation. Again, these are likely iterative processes no matter which subskill goes first.
One thing AI does very well is to propose an outline if you are able to generate a prompt describing your goals. You could then simply ask the AI service to generate something based on this outline, but this would defeat the entire purpose of learning about the topic by doing the research to translate the outline into a product or developing writing skills by expanding the outline into a narrative yourself.
Since I am writing about how AI might perform some of the subskills identified by the writing process model, I asked ChatGPT to create an outline using the following prompt.
“Write an outline for ways in which ai can be used in writing. Base this outline on the writing subprocesses of the writing process model and include examples of AI services for the recommended activity for each outline entry.”
The following shows part of the outline ChatGPT generated. I tend to trust ChatGPT when it comes to well established content and I found the outline although a little different from the graphic I provided above to be quite credible and to offer reasonable suggestions. As a guide for writing on the topic I described, it would work well.
I had read that AI services could generate concept maps which would offer a somewhat different way to identify topics that might be included in a written product. I tried this several times using a variety of prompts with ChatGPT’s DALLE. The service did generate a concept map, but despite making several follow-up requests which ChatGPT acknowledged, I could not get the map to contain intelligible concept labels. Not helpful.
Translation
Tools for improving the translation process have existed in some form for a long time. The newest versions are quite sophisticated in providing feedback beyond basic spelling and grammatical errors. I write in Google docs and make use of the Grammarly extension.
I should note that Grammarly is adding AI features that will generate text. Within the perspective I am taking here I have some concerns about these additions. Since I am suggesting that writing subskills can be replaced or supported, student access to Grammarly could allow writing subskills the educator was intending students to perform themselves to be performed to some degree by the AI.
If you have not tried Grammarly, the tool identifies different types of modifications the tool proposes different modifications the writer might consider changing (spelling, missing or incorrect punctuation, alternate wording, etc.) and will make these modifications if the writer accepts the suggestion. The different types of recommendations are color-coded (see following image).
Revision
I am differentiating changes made while translating (editing) from changes made after translating (revision). Minor changes such as spelling and grammar would seem more frequently fixed as edits by this distinction and major modifications made (addition of examples, restructuring of sections, deletion of sections, etc.) while revising. Obviously, this is a simplistic differentiation and both types of changes occur during both stages).
I don’t know if I can confidently recommend a role for AI for this stage. Pre-AI, one might recommend that a writer share their work with a colleague and ask for suggestions. The AI version of Grammarly seems to be moving toward such capabilities. Already, a writer can ask AI to do things like shorten a document or generate a different version of a document. I might explore such capabilities out of curiosity and perhaps to see how modifications differ from my original creations, but for work that is to be submitted for evaluation of writing skill would that be something an educator would recommend?
I have also asked an AI tool to provide an outline, identify main ideas or generate a summary of a document I have written just to see what it generates. Does the response to one of these requests surprise me in some way? Sometimes. I might add headings and subheadings to identify a structure I thought was not as obvious as I had thought.
Conclusion:
My general point in this post was that questions of whether learners can use AI tools when assigned writing tasks should be considered in a more complex way. Rather than the answer being yes or no, I am recommending that learning to write and writing to learn are based on subprocesses and the AI tool question should be considered in response to a consideration of whether the learner was expected to be developing proficiency in executing a subprocess. In addition, it might be important to suggest that learning how to use AI tools could be a secondary goal.
Subprocess here were identified based on the Writing Process Model and a couple of suggestions were provided to illustrate what I mean by using a tool to drastically reduce the demands of one of the subprocesses. There are plenty of tools out there not discussed and my intention was to use these examples to get you thinking about this way of developing writing skills.
References:
Bereiter, C., & Scardamalia, M. (1987). An attainable version of high literacy: Approaches to teaching higher-order skills in reading and writing. Curriculum inquiry, 17(1), 9-30.
Flower, L., & Hayes, J. R. (1981). A cognitive process theory of writing. College composition and communication, 32(4), 365-387.
Gillespie, A., Graham, S., Kiuhara, S., & Hebert, M. (2014). High school teachers use of writing to support students’ learning: A national survey. Reading and Writing, 27, 1043-1072.
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
You must be logged in to post a comment.