One of the big issues in K-12 education (and maybe higher ed. too?) during the pandemic response is assessment. Many teachers use “forced choice” assessment items (multiple choice, etc.) to assess students, and those item formats are tough to use when students are learning from home. We can try all the fancy technology we have (e.g. lockdown browsers, Hapara, countdown clocks, etc.) but no technology is likely to compete with a student, their phone, and fast thumbs. 

I worked with some teachers in my district on another option called “think-alouds,” and it’s a technique that might work for many teachers in many contexts. Here’s the general idea: instead of asking students to just choose the best answer to a multiple choice item (which a student might just google), ask them to choose the best answer AND write out their REASONING behind that answer. Require students to “think aloud” – if students represent why they think the right answer is the right answer, we can at least use that multiple choice item and the student’s writing to measure if they are thinking about the question in the ways we want them to (even if they did just google the answer). 

This isn’t a new idea: large-scale test developers often use the “think-aloud” technique to confirm that multiple choice items are measuring what they are supposed to measure. Some great folks from the Stanford History Education group used this technique to gather evidence that the items on the NAEP test that are supposed to measure historical thinking probably don’t. I used to use this technique in my psychology class as a “test correction” technique. 

Here’s a possible “workflow” for think-alouds that might work for a teacher who is working with students remotely:

1. The teacher looks at all the multiple choice items on their existing exam (the one they would use if students were in class face to face)
2. The teacher keeps all the “essential” multiple choice items and gets rid of some of the less important ones (rationale: using the think-aloud technique will take students more time, so you may need to reduce the total number of items)
3. The teacher re-formats the exam to include the multiple choice item AND space for the student to write out their thinking (e.g. a box in google docs/Microsoft word by each item, etc.) OR the teacher gives students instructions about how to add the think-aloud to the existing copy of the exam (e.g. teaching students how to add text to a .pdf using Kami, etc.- see example below) 
4. The teacher sends each student a copy of the test with a deadline. Students choose the best answer to each item AND write out WHY they think this answer is the best answer. The test directions explain this process to the students, including the message to students that the think-aloud has to be THEIR thinking, not copied from anyone or anything else (that will be obvious to the teacher if students collaborate b/c their reasoning/wording will be very similar). 
5. Students complete the think-aloud assessment, the teacher scores the tests, and sends feedback to students (either grades or feedback to students that they use to change their responses before grading). 

Here’s an example – this teacher (thanks Jim D.!) taught his students to use Kami to do think-alouds on a .pdf document.

Limitations of think-alouds:

– Students can still “cheat” – they can still look answers up on the web. But with the think-aloud technique, at least teachers can see if students are thinking through those answers correctly. 

– In assessment terms, the think aloud technique is probably a “modification”: it measures something different from a traditional multiple choice, face to face assessment. Teachers may want to think about whether they use data from the think aloud assessment in exactly the same way as they did traditional tests. 

– This is more work for students and teachers. It requires students to do more writing and teachers to read through student reasoning. Teachers should think critically about the numbers of multiple choice items they use with the think aloud technique. 

The think aloud technique is far from a perfect solution, but I don’t think we have many perfect solutions during the global pandemic. If you use this idea, please let me know so that I can update this blog post. Thanks in advance!

ADDITION (Sep. 2020):

I talked with a high school social studies teacher friend (Thanks Jeff B.!) and he encouraged me to add more specific examples. Here’s my attempt:

1. If your “normal” test is about 30 multiple choice items, you might spend up front time going through those items and narrow it down to about 10 “most important” items. These are items that you think are the “best” ones: they ask about the most important information, and they do it in a way that requires students to think rather than just recall a term/fact. That probably means you’re choosing “application” or “evaluation” items, not “recall” or “definition” items.
2. You might make two forms of the test: the in-person students get the “old” form – the one you usually use (30 multiple choice items). The zoom-in students get the new form of the test: the 10 “best” items with space for them to do the think-aloud technique, and directions somewhere explaining what they need to do.
3. You get all the tests back from all the students. The old form with 30 multiple choice gets graded however it usually got graded. The new think aloud form requires a teacher to look through what students wrote, BUT it’s only 10 items, and you only need to look at the the think-aloud part: it will be obvious from what they wrote whether they should get credit for the answer or not.
4. You figure out how to make sure there are = #s of points for both forms (this means that each of the think aloud items needs to be worth more than each of the “old” multiple choice items that the in person students tool. In my example, each of the 10 think aloud items could be worth 3 points, and each of the 30 old multiple choice items could be worth 1 point).

Note about grading: I think that grading the 30 “old” multiple choice items by hand might take about as long as looking through the think aloud responses to the 10 new think aloud items. But I could be wrong! (esp. if you had a way to automate scoring the old multiple choice items).

References:

Ercikan, K., Arim, R., Law, D., Domene, J., Gagnon, F. and Lacroix, S. (2010), Application of Think Aloud Protocols for Examining and Confirming Sources of Differential Item Functioning Identified by Expert Reviews. Educational Measurement: Issues and Practice, 29: 24-35. doi:10.1111/j.1745-3992.2010.00173.x

Smith, M. D. (2017). Cognitive Validity: Can Multiple-Choice Items Tap Historical Thinking Processes? American Educational Research Journal, 54(6), 1256–1287. https://doi.org/10.3102/0002831217717949