questions – Learning Aloud

Improve AI Generated Multiple-Choice Questions

In a recent episode of the EdTech Situation Room, host Jason Neiffer made a very brief observation that educators could improve the effectiveness of AI-generated multiple-choice questions by adding a list of rules the AI tool should apply when writing the questions. This made sense to me. I understood the issue that probably led to this recommendation. I have written multiple times that students and educators can use AI services to generate questions of all types. In my own experience doing this, I found too many of the questions used structures I did not like and I found myself continually requesting rewrites excluding a type of question I found annoying. For example, questions that involved a response such as “all of the above” or a question stem asking for a response that was “not correct”. Taking a preemptive approach made some sense and set me on the exploration of how this idea might be implemented. Neiffer proposed an approach that involved making use of an online source for how to write quality questions. I found it more effective to maybe review such sources, but to put together my own list of explicit rules.

My approach used a prompt that looked something like this:

Write 10 multiple-choice questions based on the following content. The correct answer should appear in parentheses following each question. Apply the following rules when generating these questions.

There should be 4 answer options.

“All of the Above” should not be an answer option.

“None of the Above” should not be an answer option.

All answer options should be plausible.

Order of answer options should be logical or random.

Question should not ask which answer is not correct.

Answer options should not be longer than the question.

I would alter the first couple of sentences of this prompt if I was asking the AI service to use its own information base or I wanted to include a content source that should be the focus of the questions. If I was asking for questions generated based on the large language content alone, I would include a comment about the level of the students who would be answering the questions (e.g., high school students). For example, questions about mitosis and meiosis without this addition would include concepts I did not think most high school sophomores would have covered. When providing the AI service the content to be covered, I did not use this addition.

Questions based on a chapter

I have been evaluating the potential of an AI service to function as a tutor by interacting with a chapter of content. My wife and I have written a college textbook so I have authentic content to work with. The chapter is close to 10,000 words in length. In this case, I loaded this content and the prompt into ChatPDF, NotebookLM and ChatGPT. I pay $20 a month for ChatGPT and the free versions of the other two services. All proved to be effective.

ChatPDF

NotebookLM

With NotebookLM, you are allowed to upload multiple files that a prompt uses as a focus for the chat. For some reason rather than including my entire prompt, I had better results (suggested by the service) when I included the rules I wanted the system to apply as a second source rather than as part of the prompt.

ChatGPT

The process works a little differently with ChatGPT. I first copied the text from the pdf and pasted this content into the prompt window. I then scrolled to the beginning of this content and added my prompt. I could then ask the service to produce multiple question samples by asking for another 10 or 20 questions. I found some interesting outcomes when asking for multiple samples of questions. Even the format of the output sometimes changed (see the position of the answer in the following two examples).

**4. According to Clinton (2019), what is a potential impact of reading from a screen on metacognition?**

(A) Increased understanding

(B) Enhanced critical thinking

(D) Improved retention

(**C**)

**7. Which skill is considered a “higher order thinking skill”?**

(A) Word identification

(B) Critical thinking (**Correct**)

(D) Basic calculation

From sample to sample, some of the rules I asked ChatGPT to use were ignored. This slippage seemed unlikely in the initial response to the prompt.

What is an important consideration when designing project-based learning activities?**

(A) The amount of time available to students

(B) The availability of resources

(D) All of the above

(**D**)

Summary

The quality of multiple-choice questions generated using AI tools can be improved by adding rules for the AI service to follow as part of the prompt to generate questions. I would recommend that educators wanting to use the approach I describe here generate their own list of rules depending on their preferences. The questions used on an examination should always be selected for appropriateness, but the AI-based approach is a great way to easily generate a large number of questions to serve as a pool from which an examination can be assembled. Multiple choice exams should include a range of question types and it may be more efficient to write application questions because an educator would be in the best position to understand the background of students and determine what extension beyond the content in the source material would be appropriate.

Use EdPuzzle AI to generate study questions

This post allows me to integrate my interest in studying, layering, questions, and using AI as a tutor. I propose a specific use of EdPuzzle, a tool for adding (layering) questions and notes to videos, be used as a study tool. EdPuzzle has a new AI feature that allows for the generation and insertion of open-ended and multiple-choice questions. So an educator interested in preparing videos students might watch to prepare for class could prepare a 15 minute mini-lecture and then use EdPuzzle to layer questions on this video and assign the combination of video and questions to students to be viewed before class. Great idea.

The AI capability was added to make the development and inclusion of questions less effortful. Or, the capability could be used to add some questions that educators could embellish with questions of their own. I propose a related, but different approach I think has unique value.

How about instead of preparing questions for students, allow students to use the AI generation tool to add and answer themselves or with peers.

Here is where some of my other interests come into play. When you can interact with AI that can be focused on assigned content you are to learn, you are using AI as a tutor. Questions are a part of the tutoring process.

What about studying? Questions have multiple benefits in encouraging productive cognitive behaviors. There is such a thing as a prequestioning effect. Attempting to answer questions before you encounter related material is a way to activate existing knowledge. What do you already know? Maybe you cannot answer many of the questions, but just trying makes you think of what you already know and this activated knowledge improves understanding as you then process assigned material. Postquestions are a great check on understanding (improving metacognition and directing additional study) and attempting to answer questions involves retrieval practice sometimes called the testing effect. For most learners, searching your memory for information has been proven to improve memory and understanding beyond what just studying external information (e.g., your notes) accomplishes.

I have described EdPuzzle previously, here are some additional comments about the use of the generative question tool.

After you have uploaded a video to EdPuzzle. You should encounter the opportunity to edit. You use edit to crop the video and to add notes and questions. The spots to initiate editing and adding questions are shown in the following images. When using AI to add questions, you use Teacher Assist – Add Questions.

After selecting Add Questions, you will be given the option of adding Open ended or Multiple Choice questions. My experience has been that unless your video includes a good deal of narration, the AI will generate more Open Ended than Multiple Choice questions. If you want to emphasize MC questions, you always have the option of adding questions manually.

Responding to a question will look like what you see in the following image. Playing the video will take the student to the point in the video where a question has been inserted and then stop to wait for a response.

When an incorrect response is generated to a MC question, the error will be identified.

EdPuzzle allows layered videos to be assigned to classes/students.

Anyone can explore EdPuzzle and create a few video lessons at no cost. The pricing structure for other categories of use can be found at the EdPuzzle site.

One side note: I used a video I created fitting the potential scenario I described of an educator preparing content for student use. However, I had loaded this video to YouTube. I found it difficult to download this video and finally resorted to the use of ClipGrab. I am unclear why I had this problem and I understand that “taking” video from some sources can be regarded as a violation of copyright. I know this does not apply in this case, but I did not want to mention this issue.

References:

Pan, S. C., & Sana, F. (2021). Pretesting versus posttesting: Comparing the pedagogical benefits of errorful generation and retrieval practice. Journal of Experimental Psychology: Applied, 27(2), 237–257.

Yang, C., Luo, L., Vadillo, M. A., Yu, R., & Shanks, D. R. (2021). Testing (quizzing) boosts classroom learning: A systematic and meta-analytic review. _Psychological Bulletin_, _147_(4), 399-435.

Question stems

Question stems offer guidance educators and learners can use to guide thinking about content. Becoming aware of prepared question stems can diversify the types of questions teachers ask of students and guide students in asking questions of themselves. Questions offer an external mechanism to encourage productive thinking.

Here are several links related to this video.

Learn more about the online service Insert Learning

Online lists of question stems – Question stems for critical thinking; Question stems organized according to the levels of Bloom’s taxonomy

M	T	W	T	F	S	S
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30