Category Archives: Uncategorized

Whatever AI is, it sure ain’t human

I first learned how Large Language Models work from this provocatively titled article: ChatGPT is Bullshit. When I saw the title, I clicked through because I thought it would be a fun, “joke” article, but what I found was a serious attempt to explain the non-intuitive (to me) way that so called “artificial intelligence” large-language models like ChatGPT work. I was stunned (and angered) to learn that large-language models don’t even try to “understand” or determine “meaning” from text: they use complex statistical procedures to analyze large blocks of text and determine likely large blocks of text. They are fancy text prediction. That’s it.

It might be important and useful to help students and others understand this underlying process, so I decided to make this explainer/discussion starter as a possible way teachers, etc. can start conversations about what so called “Artificial Intelligence” is. The title reveals my conclusion, so people who use this may want to change the title of the doc: Whatever AI is, it sure ain’t human (that link should ask you if you want to make a copy you can edit).

Reference: Hicks, M.T., Humphries, J. & Slater, J. ChatGPT is bullshit. Ethics Inf Technol 26, 38 (2024). https://doi.org/10.1007/s10676-024-09775-5

You keep using the term “Artificial Intelligence.” I don’t think it means what you think it means

image credit: https://en.wikipedia.org/wiki/Catch_Me_If_You_Can#/media/File:Catch_Me_If_You_Can_2002_movie.jpg

In the 2002 movie Catch Me If You Can, the character played by Leonardo DiCaprio, Frank, is impressive: he successfully pretends to be an airline pilot, a Secret Service agent, a medical doctor, and is beginning to figure out a way to fake his way through the bar exam before he goes on the run from the FBI. Frank has many impressive skills that allow him to successfully pretend in these professional situations, but he doesn’t know how to fly a plane, protect the president, cure a disease, or practice law. He’s just incredibly skilled at pretending. 

Some of the current discussion about so-called “artificial intelligence” reminds me of Frank. The term “artificial intelligence” is misleading: the most commonly used “AI” tools are more properly called “large language models.” These tools generate text in response to queries. If you’ve tried a tool like ChatGPT, you were probably getting text back in response to a question or request. That’s a large-language model. 

I think applying the term “artificial intelligence” to large-language models is an example of anthropomorphization. We seem to want (and others want us to) imagine that ChatGPT is “thinking” and coming up with “creative” text based on our query. It’s not. 

The best and most detailed description I’ve seen of how a large language model works is this explanation by Mark Riedl: A Very Gentle Introduction to Large Language Models without the Hype. Mark walks through how large-language models do what they do step by step, and ends with a great summary: 

“Large language models, including ChatGPT, GPT-4, and others, do exactly one thing: they take in a bunch of words and try to guess what word should come next. If this is “reasoning” or “thinking” then it is only one very specialized form…Our first instinct when interacting with a Large Language Model should not be ‘Wow, these things must be really smart or really creative or really understanding.’”

Large-language models aren’t doing anything like what a human brain does as it processes a question and tries to come up with an answer. What ChatGPT is doing is a verbal version of the faking that Frank does in Catch Me If You Can. It shouldn’t be called artificial intelligence because the process isn’t close to how we use the term “intelligence” in humans and other animals. It’s a process that could be called something like “statistical test prediction.” It’s a complex version of the predictive text feature many of us already use on our phones or word processing apps. A large-language model uses a huge database to figure out what word or words are likely to come next in a sentence, and it strings the most probable verbal combinations together. Here’s how cognitive psychologist and neuroscientist Hernán Anlló defines large-language models:

“Language models are AIs capable of parsing human language, and responding to it with self-generated coherent bouts of human language. Their technique is as straightforward as it is powerful: language models simply predict, based on context, how likely it is for a word to be followed by another word within a sentence, and follows this probability tree to generate its own linguistic output. Incredibly, it is by virtue of this sole operation that LLMs have managed to produce the dashing performances we have come to know in the present date.” (“Do Large Language Models reason like us?”) 

This is a complicated version of “faking” words, sentences, and paragraphs that sound good, are often accurate, and are generated quickly. It’s impressive! If you’ve played with ChatGPT, you’ve seen how quickly it can produce long stretches of text that are seemingly accurate (sometimes eerily accurate) responses to just about any question you ask it. 

But it’s not intelligence, and it’s not what we usually call “thinking.” All this technology does is produce a “coherent bout of human language” using statistical probabilities regarding the next most likely words and phrases in a sentence. It’s impressive, it’s fast, and it may even be useful in some contexts, but it’s not intelligence and we should stop calling it that. Large-language models don’t think about the meanings of words or reason through problems deductively or inductively. Some well-meaning (I think) folks inside and outside of the education community are anthropomorphizing large-language models (like this well produced video that refers to AI as a “massive external brain,” and others who call AI products “creative.”) Humans like to anthropomorphize all sorts of things (see “Is AI the Smile on the Dog”), even when this technology makes ridiculous mistakes: when a large-language model makes a mistake, developers call it a “hallucination.” It’s not! It’s a predictable result of the technology. It doesn’t know how to think or reason, so sometimes it produces results that make zero sense or even makes up sources that don’t exist

Large-language models don’t reason, see connections, or form and reform concepts or schemata. They fail at authentic reasoning tasks like the NYT “Connections” game (note: I often fail at that game too, but I fail for very different reasons than ChatGPT fails).

Some authors go beyond calling AI “fake”: I love this article a friend gave me (hey Chris P.!). I’ve been thinking about the implications of this article quite a bit, and I think the author’s bold claim, and new title for large-language models, might be justified. Don’t be put off by the title – the authors use the term “bullshit” for important reasons: “ChatGPT is bullshit.”

Why does it matter if we anthropomorphize large-language models and call them “artificial intelligence?” Because this kind of misrepresentation leads us to forget that we’re NOT dealing with “someone” or an intelligence when we interact with a large-language model, and that mistake may have consequences. In my field, education, there are some enthusiastic people breathlessly advocating for “AI everything” in teaching and learning. Large-language models might be useful in some aspects of teaching and learning, but we need to remember that they aren’t thinking like students and teachers, and learning requires thinking. If students “off-load” necessary cognition to large-language models, they might be able to produce convincing verbal products, but that ain’t learning. 

I like this rule proposed by Allison Seitchik

“The most important thing to note is that if you cannot validate AI’s response and information yourself, you cannot use it. You MUST validate and check every piece of information. In other words, if you cannot do the task yourself and check, don’t ask AI to do it.”

Students and teachers have to know how to do the tasks so that they can check the quality of the “coherent bout of human language” produced by the large-language model. Teachers and students have to learn and become experts in tasks related to teaching and learning before off-loading parts of those tasks to large-language models. If we stop calling these technologies “artificial intelligence” and start using more accurate terms like large-language models or “likely text generators,” we are more likely to remember that we’re dealing with an impressive predictive text generator, not a thinking intelligence. 

References

[2406.11012] Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game. (2024, June 16). arXiv. Retrieved June 28, 2024, from https://arxiv.org/abs/2406.11012

Anlló, H. (2024, June 3). Do Large Language Models reason like us? Research Communities by Springer Nature. Retrieved June 28, 2024, from https://communities.springernature.com/posts/do-large-language-models-reason-like-us

Busch, B. (n.d.). Can AI give better feedback than teachers? InnerDrive. Retrieved June 28, 2024, from https://www.innerdrive.co.uk/blog/ai-vs-teacher-feedback/

Greene, J.A. (2024) Is AI the Smile on a Dog? https://bemusings.ghost.io/ai-objectivity-as-a-smile-on-a-dog/ 

Hicks, M.T., Humphries, J. & Slater, J. ChatGPT is bullshit. Ethics Inf Technol 26, 38 (2024). https://doi.org/10.1007/s10676-024-09775-5 

Kniberg, H. (2024, January 20). Generative AI in a Nutshell – how to survive and thrive in the age of AI. YouTube. Retrieved June 28, 2024, from https://www.youtube.com/watch?v=2IK3DFHRFfw

Messeri, L., Crockett, M.J. Artificial intelligence and illusions of understanding in scientific research. Nature 627, 49–58 (2024). https://doi.org/10.1038/s41586-024-07146-0

Nosta, J. (2024, March 20). Is AI More Creative Than Humans? Psychology Today. Retrieved June 28, 2024, from https://www.psychologytoday.com/us/blog/the-digital-self/202403/is-ai-more-creative-than-humans

Riedl, M. (2023, April 13). A Very Gentle Introduction to Large Language Models without the Hype. Mark Riedl. Retrieved June 28, 2024, from https://mark-riedl.medium.com/a-very-gentle-introduction-to-large-language-models-without-the-hype-5f67941fa59e

Seitchik, A. E. (2024, April 30). AI research and writing companion: A student’s handbook for leveraging technology. OER Commons

Assessment is Uncomfortable

When I was a new teacher, I used what I now call the “oh crap” model of assessment: I would teach for several days, students and I would have great conversations, and then I would realize after a while “Oh crap! I haven’t put any grades in the gradebook!” I would hurriedly make a quiz or homework assignment, get it out there to students, put those grades in the gradebook, and rush back to teaching and those great conversations. 

Eventually I wised up (at least a bit) and started planning farther ahead. I stopped viewing assessment as an annoying responsibility instead of an important part of teaching and learning. I thought about how I wanted to operationally define learning in my classes and designed assessments that gave students a chance to show what they knew and could do. But it took me too long to realize why I needed better classroom assessment habits. It should be so obvious: teachers need evidence about what students are learning. 

Source: https://www.flickr.com/photos/bgentry/3853508189/

As teachers we need to know what students are learning so that they don’t “hit that bridge” when test time comes (and, more importantly, when they need to use what they learn from our classes in their lives). Why did I ignore this obvious need as a young teacher? 

Because assessment is uncomfortable. 

I think people who have never been in charge of a classroom may not be able to relate to this teaching reality: teaching is intimidating and scary, especially as a new teacher. We don’t know if we’re “doing it right.” We may have gone through years of training, but there is no amount of training that can prepare you for the moment when you are flying solo and teaching your own classes. We desperately want to do right by our students. We care. We know there is important knowledge and skills students need to learn. Checking to see what they actually learned is important, but it’s also uncomfortable. We may end up staring at evidence that what we are doing isn’t working. It is much easier to just keep “teaching” instead of stopping to check to see what students are learning. Cognitive psychologists teach us about confirmation bias: we usually look for evidence that confirms our existing beliefs. Confirmation bias can lead us as teachers to keep cruising along, teaching in comfortable ways, instead of seeking out potentially uncomfortable evidence about what our students aren’t learning. 

Assessment Discomfort Influences How Students Study

This assessment discomfort can influence how we teach, but it also influences students. Two recent, fantastic books about how students study lay out the empirical evidence for studying methods in elegant, well supported arguments: Regan Gurung in Study Like a Champ  and Daniel Willingham in Outsmart your Brain explain the mountains of cognitive psychology evidence behind advice for students about how to study (short version: studying techniques that require more thinking result in more learning. These are both GREAT books every teacher should read!) But they also explain why students tend to avoid these effective studying methods: they are uncomfortable! It’s comforting to read a chapter, think you understood it, then re-read the chapter to make sure we “got it.” This re-reading gives us a warm, comforting feeling of “oh yeah, I remember that!” and we convince ourselves we learned something. In contrast, reading a chapter, closing the textbook, getting out a blank sheet of paper (or new google doc) and writing down everything we remember from the chapter probably feels awful. This much more effective studying method is uncomfortable: we are likely to produce evidence of how little we learned from one reading of a chapter. That stinks. It’s uncomfortable. But we learn MUCH more from this uncomfortable studying technique than the comfortable, feel good re-reading of a chapter. 

Assessment Discomfort Influences Teacher Conversations

I love talking with teachers about their teaching. Usually these conversations involve lessons they just designed, compelling stories they love to use with students, and new resources they are excited to share. Rarely do these conversations involve assessment evidence (often only when I bring it up, and this suggestion is sometimes met with uncomfortable silence). Why? Because it’s a conversation killer. If someone is telling you about a new insight they are excited about and you immediately ask “Wait, what’s your evidence that’s true?” it tends to stop the conversation. My friend (and great teacher) Casey Swanson said one of the best pieces of advice he got was when an experienced teacher said “You talk a lot about teaching. How about we talk about learning for a while instead?” Teachers like to talk about teaching. Our first impulse usually isn’t to talk about evidence of student learning, but that (potentially uncomfortable) conversation might be more important than sharing another great lesson plan. 

I learned long ago to never include the word “assessment” in any conference proposal I write. Every time I used assessment in the title of a proposal it got rejected. We want our students to overcome the discomfort caused by testing themselves because we know they learn best when they figure out what they don’t know. As teachers, we should overcome our discomfort at testing our teaching. Talking about student learning, and evidence of student learning, may start the most important conversations. 

Demos for Teachers #5: Schema theory and memory

Source: https://sites.google.com/view/efratfurst/understanding-understanding

I learned about schema theory and memory from S. Chew’s article “Having Knowledge Is Not the Same as Using It” (thanks Steve! Download available here from Researchgate) and I keep thinking about applications for teaching and learning. It’s my new favorite thing!

As Steve describes in the article, schema (mental rules we use to organize the world) influence how we encode (or don’t encode) new information about the world. After I read Steve’s article, I got curious about how I missed learning about schema theory, and I think I figured it out: for some reason, schema theory is not referenced in the context of memory in the AP Psychology curriculum (that’s my primary teaching context), but it is in the IB curriculum. This seems like an oversight! Schema theory is MUCH more useful and important than many of the details AP Psychology teachers have to teach (e.g. the difference between retroactive and proactive interference – ugh!)

In his article, Steve describes research from Bransford and Johnson (1972) that involves reading a paragraph aloud to participants and testing their memory of the information from the paragraph. Their study was designed to measure the impact of listeners having or not having a useful schema for the paragraph before they get to hear it. If you didn’t jump ahead to to the article already, you can try this for yourself: read the paragraph below and think about how confident you might be if someone gave you a pop quiz about this information:

“The procedure is actually quite simple. First you arrange items into different groups. Of course,
one pile may be sufficient depending on how much there is to do. If you have to go somewhere
else due to lack of facilities that is the next step, otherwise, you are pretty well set. It is important not to overdo things. That is, it is better to do too few things at once than too many. In the short run this may not seem important, but complications can easily arise. A mistake can be expensive as well. At first, the whole procedure will seem complicated. Soon, however, it will become just another facet of life. It is difficult to foresee any end to the necessity for this task in the immediate future, but then, one never can tell. After the procedure is completed one arranges the materials into different groups again. Then they can be put into their appropriate places. Eventually they will be used once more and the whole cycle will then have to be repeated. However, this is part of life”. (Bransford & Johnson, 1972, p. 722)

The feeling of confusion or lack of confidence you may be feeling right now may be similar to how many of Bransford and Johnson’s participants felt. Some of the participants heard that paragraph without any previous information. But others got this crucial piece of information – a schema – before hearing the paragraph: “This paragraph is about washing clothes.” Knowing that schema before hearing the paragraph impacted what they learned:

Chew, S. (2022) Having Knowledge Is Not the Same as Using It, The Teaching Professor, Dec. 12, 2022

Steve does a great job in the article discussing aspects of this study that are important for teachers: participants who knew the topic of the paragraph before hearing it did better on both the comprehension and the recall measure in the study. But participants who got that information after they heard the paragraph did no better than participants who never learned the topic! There are schema students need BEFORE they go through a learning experience in order for that experience to be useful. If students don’t have the right schema, the learning experience might be a waste of time.

I talked about this experiment with science teachers recently and it started a great conversation. I first handed out a slip of paper to each teacher (pg. 2 of this handout). The teachers didn’t realize that half of them got the version of the instruction with the schema and the other half got the version without (I dealt off the top of the pile to half the room and off the bottom to the other half). Then I read the paragraph (pg. 1 of this handout) and was going to tell them whether they were group A or group B and ask that they fill out this google form (we didn’t go through with the data collection b/c it would have been tricky to ask everyone to use devices at that moment in time).

The science teachers and I had a great discussion about the implications of schema theory for teaching and learning. They immediately understood the importance of this theory for their classrooms, but they pointed out a subtlety I didn’t think about before: they often want to give students the opportunity to experience a phenomenon as a “hook” toward the beginning of a lesson or unit, and then follow up with direct instruction, etc. about the details and terminology about the phenomenon. They noted that students absolutely need the required schema before they experience the phenomena (so that they can think about it in useful ways), but they don’t need to know EVERYTHING before they get a chance to think about a concept. In fact, experiencing the phenomena and processing it as a group with their teacher can help students cement the schema in their long term memory, which they can then recall and use as they dive into the details about the concept and learn new vocabulary and concepts. Go Schema theory!

References:

  • Bransford, J. D., & Johnson, M. K. (1972). Contextual prerequisites for understanding: Some investigations of comprehension and recall. Journal of Verbal Learning & Verbal Behavior, 11(6), 717–726.
  • Chew, S. (2022) Having Knowledge Is Not the Same as Using It, The Teaching Professor, Dec. 12, 2022

Demos for teachers #4: Task switching

Source: https://www.learningscientists.org/blog/2017/7/28-1?rq=task+switching

UPDATED POST: my friend Steve Chew sent me information about this task switching demonstration, so I’m updating this post to include two versions (thanks Steve!)

This demonstration is a fast, foolproof way to help students and teachers experience the cost of switching between cognitive tasks. Even when it feels effortless and instant, interrupting a task costs us time and cognitive effort, even if it “just takes a second.”

VERSION #1: I found this activity on the fabulous The Learning Scientists blog (here’s a google doc version). The demonstration has a few steps:

  • Ask students to get out their phones and get to the stopwatch app (or pair students up and have them time each other on the tasks).
  • Task 1 = “Count as quickly as you can from 1 through 26. Try to time exactly how long it takes you – feel free to do this 2-3 times to make sure you get an accurate time. Write down your best time”
  • Task 2 = “Recite the alphabet from A to Z as quickly as you can. Again: try to time yourself precisely. Write down your best time.”
  • Task 3 = “Now you’re going to alternate those two tasks – counting from 1 to 26 and saying the alphabet. This will sound like ‘1, A, 2, B, 3, C, etc.’ Concentrate, don’t give up, and make sure you get all the way to 26 and Z! Time yourself and write down your time.”
  • Note: alternatively, on each of the three tasks you can as students to average their times across a few trials – good data collection practice, and a review of measures of central tendency!

It will be obvious to everyone that task 3 is difficult! Try to encourage everyone to persevere and finish task 3 – switching between task 1 and task 2 takes cognitive effort and many people will be tempted to give up.

You may want to collect the times on each task and average the times in order to compare those data with the graph at the top of the page from the Learning Scientists blog. But even before you analyze the data, everyone will notice that task 3 tasks much more than double the amount of time than task 1 and task 2.

You may want to use this experience to start a discussion about why task 3 takes so much more time than task 1 and 2 combined. Teachers and students may already understand that the term “multitasking” is problematic: we can’t selectively attend to more than one cognitive task at a time in our working memory. Some people think they are better or worse multitaskers, but it’s more accurate to think about “task switching:” some of us may be faster at switching between tasks than other people.

But no matter how quickly we learn to switch between tasks, there is always a cost! During task 3, we have to switch between task 1 and and 2. Each of those “switches” takes a very short time, and then switching back takes another additional short amount of time. Adding up all the time it takes to make those “micro-switches” each time explains why task 3 takes so much time and why it takes so much cognitive effort. By themselves, tasks 1 and 2 are simple, like many of the cognitive tasks we do daily. It’s incredibly tempting to leave one of these simple tasks “just for a moment” to get another simple task done immediately. While I was writing this paragraph, I saw a zoom notification on my phone that a school principal wanted help finding a report online, so I switched from my laptop to send her a quick email. The irony of switching tasks while writing a blog about task switching exemplifies how tempting task switching is. It’s often unavoidable, but acknowledging the cost is important. Teachers can help students realize that switching their attention to their phone , even just for a moment, has a real cost in time and cognitive effort. That realization might help us resist some of these task switching temptations.

VERSION #2: I like the simplicity of the 1-26/A-Z version but this similar activity might also be useful (thanks to Pat. O’Meara for finding this one! Adapted from this video: Try the Myth of Multitasking Exercise!)

  • Get out a sheet of paper and draw three lines across it horizontally so that it looks like this:
  • Tell students you’re going to time how long it takes us as a group to write the phrase “Task switching steals time” in the top row. Turn your paper over when you are done. Ready, Go!
  • Write about how long it took the class on the board (doesn’t need to be exact). 
  • Next task: in the next row, write the numbers 1 through 23. Turn your paper over when you are done. Ready, Go!
  • Write about how long it took the class on the board (doesn’t need to be exact). 
  • In the 3rd and 4th rows, you are going to alternate between letters and numbers. In the 3rd row, you are going to write the same phrase one letter at a time. After you write the S in the 3rd row, you will write a 1 in the 4th row. You may want to demonstrate this for them on the board. Keep doing that until you are done writing all the letters in the phrase in the 3rd row and the numbers 1-23 in the 4th row. Write legibly! Ready, Go!
  • Write about how long it took the group on the board (doesn’t need to be exact) – this last task will take them significantly longer! 

VERSION #3: Steve chew figured out that this one is the original version! Steve found out that David Meyer (U. Michigan) developed this activity and used it in his classes around 2008. Dr. Meyer’s version is similar to version 1 above, but is more concise:

  • Partner students up, one time keeping and one participating, then switch roles
  • Task 1 = count from 1 to 10, then immediately say the alphabet out loud from A-J
  • Task 2 = alternate between counting and the alphabet – e.g. 1-A, 2-B. etc.
  • Data analysis: divide the time for task 2 by task 1.
  • Ask students how many have a ratio of less than 1 (meaning that task 1 took more time than task 2) – there won’t be any!
  • Gather some “ratio” data from students (either from all students or a sample of students, anonymously if needed). Point out that if a ratio is 2, that means it took 2 times as long to do task 2 than task 1. That is the cost of task switching!

Steve included this graphic from Dr. Meyer’s class, which might be a great way to extend this demonstration into a discussion of the cost of task switching during studying!

UPDATE: Peps McCrea discussed the cost of task switching in his most recent “evidence snacks” email newsletter: “Interruptions Leave a Wake.” He shares this article at the end – useful research about the task-switching cost of classroom interruptions by announcement over the intercom, etc. Thanks Peps!

Demos for Teachers #3: Deep Processing

This is my favorite cognitive psychology/teaching and learning demonstration because the resulting data is evidence of possibly the most important lesson we should take away from memory theory. As Daniel Willingham said: “Memory is the residue of thought.” One of the most important ideas we should build learning experiences on is this: the more deeply we think about something, the more likely we are to encode those thoughts into long term memory.

This is a foundational idea, but we often forget it. As students, it’s tempting to use study methods that feel good and are less effortful, but effortful studying is more likely to involve deep processing and increased learning. As teachers, it’s easy to fall prey to the curse of knowledge and fail to attend to the cognitive work students need to do to learn.

In about 15-30 minutes, this depth of processing demonstration can help “prove” the learning value of deep processing by gathering data “live” in a class or professional learning session. Here’s a more complete write up of the activity – Depth of Processing

  • Look through the slides for the activity to make sure the directions make sense, and make sure everyone has a way to write down responses (paper/pencil or digital response)
  • By the end of the activity, everyone will have a list of words they remembers from the 20 “A/B” word list, and they will have “scored” how many A words and B words they remembered.
  • Ask participants to report their number of A and B words somehow (recording the distributions on a white board or google form)
  • At this point it should be pretty obvious that participants remembered more B words than A words (usually about 2 more words on average).

As you discuss with participants why we remember more B words than A words, the point of the activity should emerge out of the data: we remember more B words because the activity requires participants to deeply process B words, while A words are only shallowly processed.

When I get to discus this activity with teachers and students, we often gradually get to a simultaneously mundane and profound realization: the “harder” we think about something, the more likely we are to learn it. While this shouldn’t be a surprising conclusion, it starts great conversations about studying and teaching. If we accept this basic, foundational fact, then one of our most important jobs as students becomes figuring out how to deeply process what we need to learn. How to pick study methods that “force” us to deeply process information. And our job as teachers becomes crafting learning experiences that increase the chances that students will do the deep processing cognitive work that is likely to result in encoding to long term memory.

References:

Demos for teachers #2: Close your Eyes…

Image souce: the great Oliver Caviglioli https://www.olicav.com/#/posters/

This is the demonstration I keep in my “back pocket” and break out when I cover a class for another teacher. It doesn’t require any equipment and it’s bomb proof: it works every time, and you can use it to start discussions about hypothesis testing, or just get students cognitively engaged in thinking about how their own memory systems work.

The demonstration can also start super useful discussions with students about how they might do more useful cognitive work in class, and with teachers about how they might communicate ideas with students more effectively. It is a replication (sorta) of research done by Alan Baddeley. see the more complete write-up of the demonstration here – Baddeley’s Three Systems of Working Memory . Here’s a brief outline:

  • Ask students to close their eyes.
  • Instruct them to mentally count (don’t count out loud) the number of windows in the place where they live. They can open their eyes when they are done.
  • Ask them to close their eyes again and say “Please count the number of words in the sentence that I just said.” Repeat that sentence a couple times.
  • As they try to do that task, you might see them counting on their fingers. That’s a good sign.
  • When most of them are done, ask them to open their eyes.
  • Ask “How many of you used your fingers when you counted the windows?” One or two people might raise their hands (often no one will).
  • Ask “How many of you used your fingers when you counted the words?” Almost everyone will raise their hands.
  • Start a discussion about why that happened. These are two very similar mental counting tasks. Why did almost all of us use our fingers on the word task but not the windows task?

At this point you can list proposed explanations (hypotheses) on the board and alter the task to test them one by one, if you have time. Some explanations may get close to Baddeley’s and other cognitive psychologists’ findings: it turns out that working memory is a complex system with important “moving parts.” I like to think of these aspects of working memory as a boss and two employees (the great graphic by O. Caviglioli at the top of this blog post might help visualize these employees):

  • The boss is the “central executive” – this part of working memory monitors incoming information/stimuli and figures out what to do with it.
  • The visuospatial sketchpad employee: deals with images
  • The phonological loop employee: deals with numbers or words.

The specialties of your two employees explain this strange difference between these mental counting tasks. During the “count the windows” task, the central executive tells the visuospatial sketchpad employee to picture each window, and the phonological loop employee counts each one. Easy! No fingers needed! BUT during the “count the words ” task, the central executive needs someone to repeat the sentence that was just said since the sentence is made up of words. The phonological loop employee is perfect for that task. But now the central executive has a problem: the visuospatial sketchpad employee can’t count! So almost all of us need to use our fingers to count the words in the sentence. (Note: if you’d like to read more about this research, check out Dual Coding theory)

Cool, right? But it’s more than just cool: this detail about how our working memory operates implies some potentially important lessons for students and teachers:

Implications for studying: since we can’t concentrate on words coming in from two different sources at the same time, we may want to be careful about taxing our “cognitive load” for incoming verbal information. This may be why many of us find it difficult to read or process words while there is music with lyrics playing in the background. But the potentially more important implication for studying is that we may want to use graphics or diagrams as we study and try to make sense of words in our notes or a textbook. We can use our visuospatial sketchpad (images/diagrams) and our phonological loop (words and numbers) simultaneously to deeply process meanings and make them easier to recall later. Check out this Learning Scientists blog post for great “dual coding and studying” ideas.

Implications for teaching: After reading dual coding theory, I realized to my horror that one of my teaching habits is a BAD idea: I used to project slides with a bunch of words on them, and I would then talk over the slide, including examples of the concept, elaborating on the ideas, etc. I thought I was helping, but what I was probably doing for most students is overloading their phonological loops. Many students were probably trying to process the words on the slide (in their working memory, using their phonological loop), AND trying to understand what I was saying at the same time, overloading the same working memory system. Oops. Now I try to include only a few words on a slide (or an image) and then I talk through the concept, so that students’ phonological loops and working memories stand a better chance at processing the concept we’re discussing. Related idea: Just add blank slides.

Demos for teachers #1: Tappers and listeners

I love talking with teachers and administrators about cognitive psychology research and how these findings relate to teaching, learning, and assessment. One effective way to start these discussions is to use a “demonstration” – reproducing (at least in a limited way) a key study related to a cognitive psychology concept, and talking about what the data mean about how we think and learn.

This first demonstration – Tappers and Listeners – demonstrates the overconfidence effect. I have a more complete write up of the demonstration here – – but here’s a brief summary:

  • Students work in pairs – one student is the tapper and the other is the listener.
  • Tappers think of a very common song. They get a minute to communicate the song to the listener by tapping the rhythm.
  • Before the task, tappers estimate the % chance the listener will guess the song. Record each estimate.
  • After the task, calculate the % of listeners who successfully guessed the song.
  • The resulting data will most likely show evidence of overconfidence. The tappers’ prediction of success will be significantly higher than the actual success rate. When I use this demonstration with large enough groups of (15+), the tappers group is usually about 60-80% confident in their success, and they are only about 30-50% successful.

This demonstration is a mini-replication of E. L. Newton’s dissertation research . Generating and seeing data of the overconfidence effect “live” with a group of teachers is a great way to start discussions about how overconfidence may impact teaching and learning:

Overconfidence and studying: students are often overconfident “study-ers.” Many students use less effective study methods (like re-reading notes or a section of a textbook repeatedly) and are (over) confident that the study method will help them recall the material. This overconfidence may prevent them from trying more effective study methods (like free recall/retrieval practice). (Reference: Roediger and Karpicke, 2006.)

Overconfidence and teaching: this demonstration can also lead to a potentially important conversation about teaching. The overconfidence effect and this demonstration support the conclusion that humans are overconfident about many of our predictions. Teachers are human (last time we checked) so we should probably admit that the overconfidence effect probably leads us to some inaccurate estimates, including our perceptions about whether students are learning what we intend. If teachers are in some sense, “tappers,” and if tappers are overconfident, then we should remember that it’s likely that we overestimate how many students are “hearing the song” we are tapping. If we “go with our gut” about whether a class is “with us,” we may be fooling ourselves. This admission might lead to a discussion about the importance of quick, instant ways of checking for understanding from all students.

References:

Newton, E. L. (1990). The rocky road from actions to intentions. (dissertation).

Roediger, H. L., & Karpicke, J. D. (2006). Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention. Psychological Science17(3), 249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x

Changing my mind – grading practices

T. Guskey is one of my favorite grading authors/thinkers/researchers. His advice is clear, backed up by research, and useful for teachers. So I got interested when I saw that he co-authored a paper about standards based grading: Is Standards-Based Grading Effective?

In this article, Guskey and Link address several important topics about standards based grading (SBG) including:

  • What is Standards Based Grading? Their 3 essential criteria make sense to me. SBG = reporting on specific standards rather than a single course grade, reporting using a limited # of performance categories (rather than a % scale), and reporting academic achievement separate from behavioral categories.
  • Does SBG improve achievement? Love this quote:”No grading system by itself improves student learning.” but “better grading can provide clearer and more accurate information on students’ learning that can then be used as a basis for making improvements.” pg. 3
  • What does SBG say about classroom assessment practices? They include great reminders that classroom assessments always need to be aligned well with instruction, in SBG and non-SBG contexts. I like the distinction they make between SBG and some more general grading practices and advice that some people associate with SBG.

One specific pieces of advice in the article contradicted something I thought I knew: I suspected that if a school/district wanted to move towards SBG, they need to change their report card format at the same time. My thinking was that there needed to be a place to report standards based grades in a clear way, and that meant that schools/districts needed to change the report card format from a traditional A-F percentage based report card to a more standards based report card.

Guskey and Link directly challenge my assumption! “Although seemingly counterintuitive, reporting practices should be the last thing to change with SBG… teachers cannot report on standards until everything else aligns with them… the curricular, instructional and assessment practices used to generate these grades must be clearly addressed alongside grading and reporting policies to affect student performance. Problems arise when educators change grading practices and move ahead with SBG and reporting without addressing the critical components of curriculum, instruction, and assessment first.” (pg. 5)

So I need to keep thinking about this! When I talk with teachers who are interested in SBG, our conversations will be more about “addressing the critical components of curriculum, instruction, and assessment first,” rather than worrying about limitations of the current report card format!

Too short? Or just right? 

Lately I’m wondering about what kinds of resources might be most useful as I talk with teachers about the science of learning and memory theory. Specifically: is shorter sometimes better? 

There are plenty of great, longer explanations of how memory research relates to teaching and learning. I collected some of my favorites in this “choose your own adventure” style slide deck. That collection includes the books that started the ball rolling for me: Make it Stick  and How we Learn. Reading those books flipped on a light bulb for me: I taught memory theory every year in my psychology classes, but until I read those books, I didn’t make the connection between what I was teaching and how I was teaching. 

Those books got me started reading everything I could get my hands on about memory theory, cognitive psychology, and teaching and learning. The more you look, the more you’ll find. My most recent favorite finds are Peps McCrea’s great, tiny books (like Memorable Teaching), articles about cognitive load theory (like Cognitive load theory: Research that teachers really need to understand), and researchers who explicitly try to tie the big ideas from cognitive psych. To their teaching (like Stephen Chew’s The world’s best 24 minute videos for teachers and administrators – my title, not his!) 

But not many teachers have the time to dive into the literature like this. Lately I am getting interested in whether some VERY short presentations of the science of learning might be more effective. I stumbled across these two articles and I’m thinking about asking some teachers to read them and talk about how the ideas might relate to their teaching: 

10 Things To Know About Teaching And Learning by Alex Quigley. This blog post is simply a list of 10 statements about teaching and learning with references to psychological principles. It doesn’t get much more clear than just listing a statement like “We need to make helpful connections between the complex stuff” and pointing readers toward “schema building” (the term “deep processing” or “semantic encoding” could have worked here too). 

How Cognitive Load Theory Changed my Teaching by Zach Groshell. I love how Z. Groshell shows how he uses cognitive load theory to make practical decisions as a teacher in very short paragraphs. There are good book-length resources that use this same idea (like Small Teaching and How I Wish I’d Taught Maths) but these few paragraphs show one teacher’s rationale clearly. 

One thing I worry about: teaching and learning are so complex and contextual that any teacher could find exceptions to the “rules” or statements in these short blog posts. It would be easy to quibble with the concise conclusions in these posts. Books and longer articles have space to discuss nuances and details about cognitive psychology. I wonder: are these short summaries more prone to “lethal mutations” of what we should learn from research? 

I’m excited to see whether these very short summaries of cognitive psych. research start useful conversations with teachers. I’ll report back, and if any of you have tried anything similar, I’d love to hear about it.