Critique of Paper: An astonishing regularity in student learning rate

It rests on a critical assumption that the amount of learning that occurs during initial instruction is zero or otherwise negligible, which is not true.

The paper An astonishing regularity in student learning rate has been making rounds lately. I’ve been asked about it multiple times – the result is counterintuitive for most people, but they’re not able to pinpoint a specific issue with the setup, data, or interpretation of the experiment. The purpose of this post is to suggest a specific issue.

The Critique

The paper finds that students have wildly different baseline knowledge after initial instruction. Immediately after being shown how to do a problem, some students are almost at mastery right away and only need a couple practice problems. Other students need many more practice problems. However, during those practice problems, students’ knowledge increases at similar rates. The paper interprets this to mean that students come in with wildly different prior knowledge, but learn at similar rates.

I don’t think that’s the right interpretation. It rests on a critical assumption that the amount of learning that occurs during initial instruction is zero or otherwise negligible. But that assumption just doesn’t make any sense to me, having worked with lots of kids across different ability levels. Even if they have exactly the same prior knowledge, some kids just get it right away after you demonstrate just one instance of new skill, whereas other kids need lots of different examples and plenty of practice with feedback before they really start to grok it.

Concrete Illustration

Here’s a concrete illustration using numbers pulled directly from the paper (the 25th and 75th percentile students in Table 2). Suppose you’re teaching two students how to solve a type of math problem.

• Student A gets it pretty much immediately and starts off at a performance level of 75% (i.e. their initial knowledge level is such that they have a 75% chance of getting a random question right). After 3 or 4 practice questions, their performance level is 80%.
• Student B kind-of, sort-of gets it and starts off at a performance level of 55%. After 13 practice questions, their performance level reaches 80%.

This clearly illustrates a difference in learning rates, right? Student A needed 3 or 4 questions. Student B needed 13. Student A learns faster, student B learns slower.

Well, in the study, the operational definition of “learning rate” is, to quote, “log-odds increase in performance per opportunity . . . to reach mastery after noninteractive verbal instruction (i.e., text or lecture).” Opportunities mean practice questions. Log-odds just means you take the performance $P$ and plug it into the formula $\ln \left( \frac{P}{1-P} \right).$

• Student A's log-odds performance goes from $\ln \left( \frac{0.75}{1-0.75} \right) = 1.10$ to $\ln \left( \frac{0.8}{1-0.8} \right) = 1.39.$ That's an increase of 0.29, over the course of 3 to 4 opportunities (let's say 3.5), for a learning rate of 0.08.
• Student B's log-odds performance goes from $\ln \left( \frac{0.55}{1-0.55} \right) = 0.20$ to $\ln \left( \frac{0.8}{1-0.8} \right) = 1.39.$ That's an increase of 1.19, over the course of 13 opportunities, for a learning rate of 0.09.

So… according to this definition of learning rate, students A and B learn at roughly the same rate, about 0.1 log odds per practice opportunity.

Extreme Counterexample

In my experience, exceptionally fast learners can sometimes pick up on things so quickly that if you just show them a question that is “beyond, but not too far beyond” for them, then they will figure it out on the fly without even seeing a demonstration.

• For instance, I've worked with kids who, having learned only arithmetic but no algebra, were able to figure out how to solve the following question on the fly: "if $x$ represents a number, and $2 \ast x+3$ equals $11,$ then can you tell me what $x$ is?"
• And I've also worked with students who have no issues with arithmetic but struggle to solve that kind of question even after seeing a demonstration of how to solve it and going through a couple practice questions with feedback.

As a specific example, I taught algebra to this really talented kid many years ago, and pretty quickly I realized that the quickest way to get him through the content was to just ask him questions like that (“if $x$ represents a number, and $2x+3=11,$ what is $x$”) without even demonstrating anything.

• Most of the time he would completely figure it out on the fly rather quickly.
• Sometimes I'd have to draw his attention to edge cases: "okay, you told me that the solution to $x^2=9$ is $x=3,$ but is there any other number that you can square and get the same result?"
• Other times I'd have to give him a tip to get him started: "okay, you need some help getting started with solving $x^2 + 2x = 9,$ see if you can add something to both sides of the equation that lets you write the left-hand side as a perfect square." (He had not previously learned the trick for completing the square).

With these kinds of students, you can often teach formulas by having them derive the formulas themselves. For example:

Of course, these kinds of students are extremely rare, but they illustrate just how incorrect it is to assume that differences in problem-solving ability after initial instruction (or even before initial instruction) indicate differences in prior knowledge.

Defense Against Misinterpretation

To be clear, I want to see every single student grow and learn as much as they can. I don’t think anyone’s level of knowledge is set in stone.

But in order to support every student and maximize their learning, it’s necessary to provide some students with more practice than others. If a student is catching on slowly, and you don’t give them enough practice and instead move them on to the next thing before they are able to do the current thing, then you’ll soon push them so far out of their depth that they’ll just be struggling all the time and not actually learning anything, thereby stunting their growth.

Likewise, if a student picks up on something really quickly and you make them practice it for way longer than they need to instead of allowing them to move onward to more advanced material, that’s also stunting their growth.

I’m 100% in the camp of maximizing each individual student’s growth on each individual skill that they’re learning, giving them enough practice to achieve mastery and allowing them to move on immediately after mastery.