Self-Reports of Learning Are Ridiculously Unreliable

Measuring performance is the only way to reliably assess knowledge and learning.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

Self-reports of learning are ridiculously unreliable. There are studies on this, and it’s also common sense to anyone who has been responsible for the learning of other people, but what really drives the point home is experiencing it firsthand.

Just tonight I was practicing memorizing some song lyrics (my sister-in-law and I have started bonding over music so I make an effort to actually know things when we talk about what she’s listening to).

• Song #1: Thought I knew it well enough to get through with 20-30 hints. Turns out I was completely wrong, I only knew two lines of the chorus and the catchy beat, but the rest of the song I couldn't even remember how the music went while staring at the lyrics. Listened to the song again and the verses and bridge popped out at me as though I had never even heard them before.
• Song #2: I did decently on this one but was disappointed at the end because I felt like I did about the same as last week. Then when I logged my hint count and saw last week's performance I realized I only used half as many hints today, 13 instead of 25.

So on one song, my feelings way overestimated my knowledge, and on the other, my feelings way underestimated my performance gain.

And this is coming from a guy who is fully aware of this phenomenon, spent years tutoring and teaching, and measures knowledge/learning for a living.

I cannot even trust my own feelings about my own learning. I have to actually measure performance.

Measuring performance is the only way to reliably assess knowledge and learning.

Follow-Up Questions

What are you using to memorise song lyrics? The hints sound interesting. I’ve tried putting them in Anki in the past but it hasn’t worked very well for me.

Very low-tech solution: I just look up the lyrics on my phone, cover the bottom half of the phone with a credit card, and scroll up, trying to predict each successive line. I simultaneously recall the melody and backing music in my head. Basically, I’m trying to replay the next “line” of the song in my head (lyrics, melody, backing music).

To measure performance, I count the number of lines I get incorrect or can’t remember how the melody/backing music goes. I called that number “hints” in the post above since when I get a line incorrect and look at it, it serves as a hint for the next line. I use a physical tally counter to track the number of hints I use as I go through the song.

I keep track of my progress in a spreadsheet and do a low-grade spaced repetition where the number of days between repetitions is a simple function of the number of hints on the previous repetition. Right now I’ve hard-coded that function as 100/(1+numHints) but that’s subject to change.

If it turns out that I don’t know the song well enough and am forgetting nearly every single line and/or can’t remember how the song goes, then I drop down to an easier form of practice where I actively listen to the song while constantly trying to predict the next few words. I also jot down lines that I like in particular and memorize those in isolation to create some memory anchors. But once I know the song well enough for the mental replay to be effective, then I switch over entirely to that.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.