Learning Rate Differences Exist

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

The existence and magnitude of learning rate differences is a hard truth that many refuse to accept because they wish it weren’t so.

There’s a recent paper that

not only measured the difference as 2x between the 25th and 75th percentile
but also did this using a methodology that systematically undershoots the difference (by assuming all differences in initial performance stem entirely from differences in background knowledge unrelated to learning rate).

Unfortunately

the authors did not acknowledge the methodological shortcoming,
they reported the 2x factor as 1.5x in rate of raw increase in expected accuracy (which is really 2x when you measure in log-odds, a more appropriate metric that accounts for the fact that it’s harder to increase accuracy when one’s accuracy is high to begin with),
and they concluded that 1.5x difference between 25th and 75th percentile is “An astonishing regularity in student learning rate” and named the paper as such,
so it inspired a bunch of media articles with titles like "the myth of the quick learner" and "quick learners don't exist,"
even though it's actually evidence to the contrary.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.