Challenges One Might Encounter While Leveraging Effective Pedagogical Techniques

by Justin Skycak (@justinskycak) on

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

Anatoly Vorobey posted a list of challenges that he experienced while tutoring a student while leveraging effective pedagogical techniques. These were interesting to evaluate and I expect they may be useful reading for any educator in a similar position.

Challenge 1

“But when in a process of solving something else (like a word problem) two unknown quantities come together in two different ways, and you get two equations, she wouldn’t jump from that to solving a system.”

Word problems would need to be a separate topic that the student gets repetitions on. There are two things going on here, 1) you can set up a system of equations describing a word problem, and 2) you can solve that system using previously learned techniques. I generally wouldn’t expect a student to naturally make the jump from (1) to (2) on their own unless they’re exceptionally bright. The vast majority of students need to be taught this stuff explicitly and it takes lots of repetitions spaced out over time before it becomes a natural reflex (kind of like muscle memory).

“They [inequalities] come up all the time in different contexts (problems with absolute value, domains of functions etc.). But knowing how to solve an inequality and knowing to recognize something as an inequality to solve are two different skills. … Solving inequalities on their own doesn’t help with this. … Solving lots of problems that depend on inequalities? If they’re all typical and basic, this doesn’t seem to train up the concept enough. Maybe I should try to assemble a *diverse* set of very different problems that hide inequality in different ways. But that seems too clever.”

Yes, assembling a diverse set of problems that hide the inequality in different ways is what needs to be done. For the vast majority of students, this is a skill that needs to be hit over and over again, spaced out over time, in each context you expect them to apply it in. For students who are particularly mathematically gifted, they will require fewer repetitions and a smaller variety of examples from which to generalize. But most students need a large spread like that where they can get plenty of explicit practice in all these contexts. Of course, this takes a ton of work to have them hit all those bases, but it’s what needs to be done.

Challenge 2

“a meta-algorithm that looks something like: you’re given some setup with lots of interrelated thingies and some facts about them, and you need to find the numeric value of one of the thingies. Denote some of the thingies as variables and express others in terms of those variables using the facts. Work towards an equation, knowing it must come. When it comes, solve it; sometimes there’ll be more than one, then solve the system. … Maybe the meta-algorithm is trainable for everyone/most people, but some need x100 fewer reps than others.”

Yes, I would agree that students who are particularly gifted can generalize from far far fewer reps, while less gifted students may need 10x or 100x or maybe even more. I actually benchmarked this once by estimating how many practice problems it took to prepare some of my students for the AP Calc BC exam – it took about 400 problems for a kid who was prodigy-level gifted, and about 4000 problems (10x more) for kids who were “just” highly gifted. For students who are “just” above average, or typical, this multiplier will shoot up way higher. This is discussed in detail here, and I’ll paste two particularly relevant paragraphs below:

  • "Midway through 7th grade, M and his parents started thinking more about his future and came to agree that it would be a good idea for him to knock out the AP Calculus BC exam in 8th grade so that he could make a convincing case for enrolling in university courses the following year. But at the same time, he wasn’t going to take a separate calculus course, and he didn’t want to do a whole lot more work outside of our weekly discussions. The totality of M’s calculus instruction before the exam was limited to an hour-long chat each weekend and 5-10 homework problems per week for about 12 months, about 400 problems total, plus 3 or 4 practice exams.

    Meanwhile, students that I taught in the radically accelerated school program did about 300 lessons (each with about 10 questions) and 300 reviews (each with about 3 questions) for a total of about 4000 questions, plus 6 practice exams. Not only did they solve an order of magnitude more problems than M student, they also had more one-on-one time with me (there were only 5 students in the class), they were doing this every single school day for an hour, and they were also working from a far more scaffolded and comprehensive curriculum (whereas for M, I had to slim down the curriculum to the bare essentials, otherwise we wouldn’t get through it all). Yet, M thought the AP exam was pretty easy, came out of the exam quite confident that he got a 5 out of 5, and indeed he did – whereas in my class, the average score was 3.6 out of 5 (two 5’s, a 4, two 2’s), and even the students who ended up getting a 5 did not come out of the exam confident that they got a 5."

Challenge 3

“3. Negative knowledge. Remembering what NOT to do, even though it seems obvious. We can solve many inequalities with a method of intervals, then after a while I throw in a simple “1/(5-x) > 4”, and her first instinct is to - obviously - turn this into 1 > 4(5-x). It seems harder to train NOT to do something than to do something right. This problem does seem to be helped by doing many very simple reps interspaced, but it remains to see if the learned negative knowledge will stick.”

It’s good to hear that interleaving between both categories of problems is helping. I would expect the learning to stick provided that you follow a reasonable spaced repetition schedule. The knowledge is always in a state of decay; there is no one-and-done solution to make it stick forever on the first exposure, but if you periodically have the student solve review problems then the rate of forgetting should gradually slow down enough that no more explicit review is needed (because the student ends up happening to practice this as a subskill in more advanced material frequently enough to keep the forgetting from getting too extreme).

Challenge 4

“4. Focused attention. So many small errors are just inattention. Algebraic manipulation especially. Repeated focus on fundamentals makes it better but it’s not clear that it makes it go away completely. It usually helps to say “re-check your work” even without pointing out the mistake, so maybe the idea is to train *that* to happen automatically.”

Small errors due to inattention should go away gradually as the student develops full mastery of the material (by getting more practice spaced out over time while also layering more advanced knowledge on top).

This won’t happen super quickly, but the rate of making errors should decrease as 1) the mathematical manipulations become almost like muscle memory, 2) the student experiences less cognitive load while solving the problems, and 3) they can better maintain a “big picture” view of what they’re doing and detect early on when something is going off the rails.

(3) in particular comes down to perceptual learning, the ability to extract key features from complex environments while filtering out irrelevant noise, which comes down to building long-term memory representations (elaborated here).

As you’ve pointed out, a critical component of developing full mastery is holding the student accountable for solving problems correctly, independently. If they are always told where their mistake is, they will not develop the ability to catch mistakes on their own, which means they will never reach full mastery of the skill.

(If a student is unable to identify their mistake then of course you’d need to provide some scaffolding by nudging them towards where they need to look, or if they’re really stuck, then you might even need to point it out explicitly – but this scaffolding needs to be gradually ripped away.)

Additionally, this is another reason why it’s critical that the student builds sufficient mastery on prerequisite skills – if there are 4 subskills involved and the student executes each of those subskills correctly 70% of the time, then the student’s probability of landing the compound maneuver correctly is only 0.7^4 = 24%. The student needs to practice their subskills enough to become really really solid executing them. Frequent silly mistakes are often a symptom of not being solid enough on component skills (i.e., not having gotten enough practice on those component skills).

Challenge 5

“5. Long-term memory. This is the obvious one. Everything fades. Specific examples are the formula for roots of the quadratic equation, and the order of terms in a derivative of a fraction (f’g - fg’, not the other way around). Spaced repetition can and does help with forgetting, but I also do wonder if different students have vastly different lengths of retention before they need to be reminded.”

Yes, different students will have vastly different retention intervals (and so will different topics depending on intrinsic difficulty). This is actually factored into Math Academy’s spaced repetition system, which I wrote about here. Here’s a particularly relevant snippet from the section “Calibrating to Individual Students and Topics”:

  • "The speed at which students learn (and remember what they’ve learned) varies from student to student. It has been shown that some students learn faster and remember longer, while other students learn slower and forget more quickly (e.g., Kyllonen & Tirre, 1988; Zerr et al., 2018; McDermott & Zerr, 2019). Similarly, learning speed also varies across topics: easier topics are learned faster and remembered longer, while harder topics take longer to learn and are forgotten more quickly.

    So, for each student, each topic has a learning speed that depends on the student’s ability and the topic’s difficulty. Student ability and topic difficulty are competing factors – high student ability speeds up the overall student-topic learning speed, while high topic difficulty slows it down. In this view, a student-topic learning speed can be measured as a ratio between
    1. the speedup due to student ability, and
    2. the slowdown due to topic difficulty.
    Student-topic learning speeds are used to adjust the speed of the spaced review process."

I also wrote about this in the post that I mentioned earlier about the prodigy-level student versus students who are “just” highly gifted. I’ll paste two particularly relevant paragraphs below:

  • "M would also retain it much longer after the initial practice – for instance, we could cover a new topic one week and then he’d be able to recall most of it a week or two later, whereas many students in my class would forget most of a new topic within a couple days of learning it if they did not receive additional practice. That said, M is not immune to forgetting, and it’s not like he’s “locking things into place” indefinitely in his brain. It’s just that his rate of forgetting is much slower.

    I’ve worked with plenty of students who are well above average mathematically but not nearly to the extent of M. They are much slower to absorb new information, and even after they are able to consistently solve problems correctly, they will forget it almost entirely within a week or two. Imagine you’re writing code to develop some application, but you’re using some buggy version control where each day, 10% of your code is deleted. That’s how it feels working with these other students. It’s like writing in disappearing ink. On the other hand, for M, it’s like his code gets implemented in a more robust way, and less than 1% gets deleted each day."

Challenge 6

“Say when solving a geometry problem with trigonometry, you get to a triangle, and you know some side lengths/angles, you need others. You can set up an equality with a law of sines, and it’s often helpful. But if your triangle is right-angle, you don’t need the law of sines. … [but] It seems counterproductive to specifically teach ‘do NOT use this in a right angle triangle’”

This strikes me as a variation of example 3 (remembering what NOT to do, even though it seems obvious), and I would expect interleaved spaced practice to help here, provided that the student is receiving corrective feedback to use the simpler formula.

One additional thing that I’d expect to be helpful here is timed practice. Many students will happily resort to always using a more cumbersome method if it means there’s less to remember, but as you point out, it’s typically counterproductive to treat an correct answer as incorrect on the basis of a non-preferred solution methodology. However, you can shut down inefficient solutions by adding a time constraint – the student legitimately can’t solve the problem within the expected amount of time, and learning to pick the more efficient solution methodology can help them overcome this. The time constraint helps frame the situation more productively, and timed practice (with follow-up practice on anything the student can’t do quickly enough) is needed anyway to help build automaticity (only after a student is able to successfully perform the skill in an untimed setting, of course).


Want to get notified about new posts? Join the mailing list and follow on X/Twitter.