Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

This FAQ is taken from The Math Academy Way.

THE PRACTICE EXPERIENCE

Active Learning, Scaffolding, and Automaticity

> How does a lesson work?

At the most fundamental level, a lesson is a sequence of slides with instructional text, interspersed with active problem-solving.

Each lesson starts out with an introduction, and then moves to a worked example, followed by 2-5 practice questions on the same type of problem as the worked example. The worked example and active problem-solving we collectively refer to as a “knowledge point” or KP.

The number of practice questions in a KP adapts to the student’s performance: practice questions continue until the student demonstrates sufficient mastery of the KP to continue building more advanced learning on top of that understanding.

After completing a KP, the student moves on to the next KP. A typical lesson has about 3 or 4 KPs increasing in difficulty. The first KP covers the simplest case to introduce a new concept/skill, and the following KPs build on this concept/skill, extending it to progressively more advanced cases.

(If a student is unable to demonstrate mastery of a KP within 2-5 questions, then they “fail” the lesson and spend some time working on other lessons before coming back to re-attempt the originally failed lesson. On average, students pass lessons on the first attempt 95% of the time and within two tries 99% of the time.)

> Solving problems breaks my flow of learning. Is it really necessary?

Active problem-solving is where the learning happens. It may feel like learning when you’re following along while reading/skimming a book, but that comfortable fluency is completely artificial. It arises from the fact that the surrounding context is already on your mind, and you’re not actually being made to pull it from memory.

If you define learning as a positive change in long-term memory, then you haven’t learned unless you’re able to consistently reproduce the information you consumed and use it to solve problems. This doesn’t happen when you just “follow along,” even if you understand perfectly. It’s the act of retrieving information from memory that transfers the information to long-term memory. If you don’t practice retrieval, then the information quickly dissipates. It stays with you only briefly – just long enough to trick you into thinking it’ll stick with you, when it’s really on the way out the door.

The most effective way to avoid this problem and maximize your learning – not just the perception of learning – is to switch over to active problem-solving immediately after consuming a minimum effective dose of information. While this may initially feel a bit jarring, it isn’t slowing down your learning – it’s only exposing the fact that your perception of learning does not accurately reflect actual learning. In reality, it’s speeding up your actual learning, and the only thing it’s slowing down is your perception of learning.

You might say “but I had learned so much, and I had it down pat, and then I forgot it all when I focused my effort on solving a problem.” But the thing is, if you can’t retrieve that information from memory at the snap of a finger, after thinking about other things or zooming in to focus on a specific problem, it means you didn’t really have it down pat. You just felt like you did because you weren’t being made to attempt to regenerate the information from scratch, from memory. What you’re really saying is “I was juggling a lot of information in my working memory (WM), and I thought it was in my long-term memory (LTM), and then I cleared a lot of it out from my WM when I focused my effort on transferring some of that WM into LTM.”

> Don’t students need to struggle for long periods of time, without too much guidance, to train their general problem-solving ability?

For students (not experts), empirical results point in the opposite direction. One key empirical result is the expertise reversal effect, a well-replicated phenomenon that instructional techniques that promote the most learning in experts, promote the least learning in beginners, and vice versa. It’s true that many highly skilled professionals spend a lot of time solving open-ended problems, and in the process, discovering new knowledge as opposed to obtaining it through direct instruction. But that doesn’t mean beginners should do the same. The expertise reversal effect suggests the opposite – that beginners (i.e., students) learn most effectively through direct instruction.

Additionally (and relatedly), as discussed in The Math Academy Way: there’s a mountain of empirical evidence that you can increase the number of examples & problem-solving experiences in a student’s knowledge base – but a lack of evidence that you can increase the student’s ability to generalize from those examples (by doing things other than equipping them with progressively more advanced examples & problem-solving experiences). In other words, research indicates that the most effective way to improve a student’s problem-solving ability in any domain is simply to equip them with more foundational skills in that domain. The way to increase a student’s ability to make mental leaps is not by having them jump further, but by having them build bridges from which to jump.

There does not seem to be any tangible, empirically-supported reason for a student to struggle with a problem for a long period of time as opposed to using that time to learn more content. For instance, in an hour-long training session, a student will make a lot more progress by solving numerous “deliberate practice” problems that each take a small amount of time given their current level of knowledge, than by attempting a single problem that they struggle with for a long period of time. (To be clear: the deliberate practice problems must be grouped into minimal effective doses, well-scaffolded & increasing in difficulty, across a variety of topics at the edge of the student’s knowledge.)

As Sweller, Clark, and Kirschner sum it up in their 2010 article Teaching General Problem-Solving Skills Is Not a Substitute for, or a Viable Addition to, Teaching Mathematics:

Another good reference is Putting Students on the Path to Learning: The Case for Fully Guided Instruction by the same authors (Clark, Kirschner, & Sweller, 2012). It’s an expanded version of the 2010 article.

> Is automaticity really required to move up to the next level? Doesn’t it just come with time?

While we emphasize the importance of building automaticity over time (see The Math Academy Way), we do not mean to suggest that students have to learn skills to the point of automaticity before moving forward.

Before moving forward, students must reach a “baseline mastery” performance threshold indicating that they have learned the material well enough to solve problems successfully (and do this consistently). This level of baseline mastery is necessary for students to continue layering on additional knowledge where baseline-mastered skills are exercised as component sub-skills within more complex skills. However, the performance threshold for baseline mastery is not as high as the performance level for automaticity, which is realized over a longer period of time.

As discussed in The Math Academy Way, layering more advanced skills is one of the most efficient ways to achieve automaticity: as students learn progressively more advanced material, they reinforce and deepen their foundational knowledge. However, the efficiency of layering is conditional on students being able to successfully execute the foundational skills, which requires a baseline level of mastery.

Furthermore, while one may hope to naturally develop automaticity on lower-level skills by layering on more advanced skills, it is still necessary to check that this is happening and take swift action if it’s not. This is one reason why Math Academy leverages frequent timed assessments and immediately follows up with remedial support on any questions a student misses.

Checking for automaticity will continue to grow as a centerpiece of the Math Academy system, especially in the context of teaching “math facts” like addition and multiplication tables, since it is easier for a lack of automaticity to fly under the radar in those areas (due to how simple and quick the problems are). To provide a concrete example: sometimes a student will default to recalculating (or even finger-counting) every single fact instead of first trying to retrieve it from memory. At first their speed and accuracy will increase because they’re getting better at recalculating, but these gains will asymptote off before the student reaches anywhere near the range of automaticity. This kind of student will never develop the necessary automaticity unless somebody intervenes to break them out of their habit and support them with flashcard-style practice. As Math Academy develops a “math facts” curriculum, these automaticity interventions will be built directly into the system.

> If worked examples are necessary to maximize learning efficiency, then why am I able to solve problems just fine without them?

When students start out learning math, it sometimes feels easy to the point that they can solve problems reasonably quickly without having to see worked examples. But this phase is temporary: as the level of math rises, solving problems without worked examples quickly becomes overwhelming and inefficient. Without worked examples, learners reach a point where unguided problem-solving overwhelms their working memory and puts them in a state of cognitive overload where they feel frustrated, confused, and are unable to solve the problem. They flat-out stop making progress, and no more learning happens.

Even before a lack of worked examples becomes a complete roadblock to successful problem-solving, it will inflate the amount of time needed for a student to successfully solve problems, thereby throttling the volume of deliberate practice cycles that can be achieved in any given amount of time. This is problematic because (as discussed in The Math Academy Way) the accumulated volume of action-feedback-improvement cycles is the single biggest factor responsible for individual differences in performance among elite performers across a wide variety of talent domains.

In summary: math gets hard for different students at different levels – it can be as early as high school algebra or as late as graduate-level Algebraic Topology – but everyone eventually reaches a level where things no longer feel obvious and they can’t figure things out as quickly on the fly. That’s where worked examples and instructional scaffolding come in to keep students making fast progress. If you don’t have worked examples and instructional scaffolding to help carry you through once math becomes hard for you, then every problem basically blows up into a “research project” for you. That’s okay if you’re a research mathematician at the edge of your field, but if you’re a student who still has a ways to go before reaching the edge of human mathematical knowledge, then it’s far less efficient (even if you have fun with it).

Of course, if you really want to solve problems without referring to worked examples, nobody is stopping you from skipping worked examples and trying your hand at solving the corresponding problems without guidance. It can be a fun challenge! You just need to make sure that you’re still solving the problems quickly and accurately – if you start slowing down and/or becoming less accurate (or, more subtly, if you start to doubt yourself and lose interest), then that’s an indication you need to start leveraging those worked examples.

> Why aren’t Math Academy’s university courses structured like typical higher math textbooks with minimal scaffolding?

Higher math textbooks and classes are typically not aligned with (and are often in direct opposition to) decades of research into the cognitive science of learning. Higher math is heavily g-loaded, which creates a cognitive barrier for many students. The goal of guided and scaffolded instruction is to help boost students over that barrier. (To be clear, we do not mean to imply that higher math would be “easy” if taught properly – just that many more people would be able to learn it, than are currently able to learn it.)

Why do higher math textbooks lack such scaffolding? For one, the amount of work it takes to create a textbook explodes with the level of guidance and scaffolding, so in practice there is a limit to the amount of boosting that is feasible, especially if the textbook is written entirely by a single author.

That said, most higher math textbooks don’t even come close to the theoretical limit for a single author, much less the theoretical limit for a team of content writers. Why is that? First, consider the following problem that has affected anyone who has ever tried to learn math from a textbook: a worked example demonstrates a special case, but a practice problem requires a logical leap that wasn’t explicitly covered. A number of textbooks seemingly attempt to solve this problem by side-stepping the need for a large amount of scaffolding (worked examples and practice problems increasing in difficulty), and instead focus the effort on trying to teach general problem-solving skills with challenging problems that require large mental leaps.

However, as discussed in The Math Academy Way, there is a mountain of evidence in the cognitive science literature that you can increase the number of examples and problem-solving experiences in a student’s knowledge base, but a lack of evidence that you can increase the student’s ability to generalize from those examples. In other words, research indicates that the best way to improve one’s problem-solving ability in any domain is simply to acquire more foundational skills in that domain. The way you increase your ability to make mental leaps is not actually by jumping farther, but rather, by building bridges that reduce the distance you need to jump.

Higher math textbooks and courses often focus on trying to train jumping distance instead of bridge-building – especially once a student gets into serious math-major courses like Real Analysis and Abstract Algebra. However, what actually works in practice is simply creating more worked examples, organizing them well, and giving students practice with problems like each worked example before moving them onto the next worked example covering a slightly more challenging case. Students can successfully climb to higher-than-expected levels of math with this approach, but many educational resources shy away from it because it takes so much work to create all the necessary content.

> I expected the quiz to cover topics that I learned since the previous quiz, but it asked me about lots of topics that I learned even before that. I would have done better on the quiz if I knew what was going to be on it or if it just limited the questions to what I’ve learned recently. Is this a bug? It feels weird and unfair.

Quizzes cover all topics a student has learned on the system, not just new topics between quizzes. The goal of quizzes is to measure a student’s level of automaticity on material that they’ve previously learned and practiced enough to expect a reasonable degree of automaticity to have developed. Quiz performance helps the system understand whether it’s moving at the right pace for a student or if it needs to slow down and provide more frequent practice on previously learned material to help the student retain it and develop proper automaticity.

If quizzes were limited to topics covered since the previous quiz, that would telegraph what’s going to be on the quiz (causing it to be artificially easy) and exclude older topics where it’s most important to be measuring automaticity. This would dilute the efficacy of the quizzes in adapting the pace of learning and promoting retention & automaticity.

We realize that in a typical classroom, quizzes tend to be less frequent, students are told what’s going to be on it, it only covers topics they’ve learned very recently leading up to the quiz, and extensive time is permitted to solve each question. But those conditions make quizzes artificially easy, a biased signal for adapting the pace of learning, a poor measurement of retention & automaticity, and an inferior tool for promoting retention & automaticity. It’s like playing a game of football where the opposing team asks what plays you’ve been practicing in the past week, and then selects their own plays so that the appropriate counter-plays are the ones you recently practiced, and then tells you what plays they’re going to run. It’s not a real game. It’s completely artificial.

We also realize that this can be a rude awakening for many students who are accustomed to less effective techniques leveraged in more typical educational offerings. We can definitely improve on helping learners understand the rationale behind these sorts of decisions made by our system, and it’s on our roadmap. But at the end of the day, the purpose of Math Academy is to ascertain the truth about what a learner knows and how well they know it, and leverage said truth to maximize learning efficiency, even if this process can lead to some initial unfamiliarity and discomfort.

> I perform well in the lessons themselves, but do not recall or recognize patterns as well when doing quizzes. Is this normal?

Quizzes are a much tougher setting than lessons since they’re timed, fully interleaved, and some forgetting has set in, so it’s expected that you might miss some questions. That’s intentional – in fact, we also calibrate the quiz difficulty to make them tougher if you’ve been doing well on them, because the 80-85% accuracy range is the sweet spot for learning.

One of the many goals of quizzes is to pinpoint areas where you are not yet fully automatic (even if you could solve those problems in an untimed open-book setting). We want to help you build automaticity, so the quizzes help us pinpoint the areas where you’d benefit from additional practice (which is provided as follow-up reviews after the quiz, and then a quiz retake if you didn’t knock it out of the park the first time around).

Over time, through quizzes, reviews, and generally layering more advanced content on top of that you learned, you’ll develop a stronger grasp of the material you covered in lessons. If you’ve been consistently using the system for several months or more, then I bet if you look back to a quiz you took several months ago, the questions you missed will feel pretty easy now.

> I am in the SAT Prep course and I’m only getting full-length practice tests every 500 XP. The rest of the time I’m doing lessons and reviews and quizzes on SAT Prep topics. Shouldn’t I be focusing entirely on full-length practice tests?

I know it may feel like you should just grind practice tests entirely instead of the usual topics / quizzes, but that isn’t really effective, because practice tests are not targeted to your points of weakness, or even the most challenging content. While practice tests are a key component, you really want to be focusing the majority of your practice on the trickier stuff.

Spaced Repetition

> After completing a course, my knowledge profile visualization shows many topics as light blue instead of dark blue. Shouldn’t these topics be closer to full mastery?

Completing a course means that all the topics have been completed (i.e., baseline mastery, enough to continue building on). Many of these topics will not be close to full mastery because you can’t reach full mastery just a day or even a week after your first exposure to a topic. Full mastery would mean you’re extremely far into your spaced repetitions, which takes months. This is not a product design choice but rather a law of cognitive physics. Baseline mastery can be achieved in a single session, but full mastery requires reinforcement over time.

> Is the spaced repetition happening? I started recently and all I have is lessons. Where are the reviews?

A lot of the review happens implicitly by having you learn new topics that encompass previously learned topics as subskills. At the beginning, when you have a small body of knowledge to review, we’re able to pick new lessons that knock out all your reviews without you having to explicitly do any review tasks. However, as you build up a larger body of knowledge to review, you’ll start to see explicit review tasks on topics that we are not able to knock out explicitly.

So, basically: the spaced repetition has already started kicking in, but we do a lot of optimization to make that happen simultaneously while having you learn new material. The only time you’ll get an explicit review is when we’re not able to knock it out implicitly while having you learn something new. (Though, after a quiz, you’ll also get explicit reviews immediately on any questions you miss.)

> I’ve been having more review tasks lately than I’m used to. What’s going on?

It’s normal to sometimes have strings of many consecutive lessons, and other times have plenty of review tasks with lessons interleaved every several reviews. Likely, you have gotten used to having the balance shifted more towards lessons, which is what happens when we are able to implicitly knock out more reviews, just because there happen to be more encompassings in that area of the knowledge graph. Now, we have a backlog of review from those lessons, and we just happen to be in an area of the knowledge graph where there are less encompassings that we can use to knock out reviews implicitly.

By way of analogy: we are a really fast car (top speed 1000 mph) and we’re traveling across a varied terrain. Sometimes the terrain permits traveling at 1000 mph. Sometimes we have to slow down to 500 mph just because the terrain has turns, potholes, etc. We’re still moving as fast as we can. But, of course, if a passenger doesn’t understand the terrain, then they might take issue when the car slows down from 1000 mph to 500 mph.

Also keep in mind that at the beginning, you get mostly lessons because there’s very little for you to review – but as you acquire more knowledge, there’s more to review. We’re always knocking out as much as we can implicitly, but yes, in the long game you’re going to get more review tasks than lesson tasks. (Not an excessive amount, though. In the long game you’re looking at a couple reviews per lesson on average.)

> I am working consistently on the system but have been doing less XP per day over break. Why are my tasks a higher proportion of reviews than I’m used to? I thought if I’m doing fewer lessons then I should be getting proportionally fewer reviews?

Every lesson you complete creates future review obligations, so whenever a student’s pace drops, it causes a wave of reviews to build up.

It’s kind of similar to what you would expect in financial situations with delayed payments. Say you charge $5k to a credit card each month and pay off $5k from the previous month. As long as the charges and payments stay balanced, everything feels stable. Now say you suddenly reduce your monthly credit card volume from $5k to $1k. You’re only charging $1k and you think that you should only have to pay off $1k. But really, you still owe $5k from the previous month, so you’re going to have to pay off more than you charged this month.

Reviews work the same way. If you are doing say 15 lessons and 30 reviews each week for a while, and then you slow down your pace a bunch, you don’t get to carry on just doing 3 lessons and 6 reviews each week. You still have lots of reviews coming due based on your prior activity. You don’t get to stop paying off old loans just because you stopped opening so many new ones. So this shows up as a wave of reviews that gradually fades as you pay off the imbalance.

This can feel particularly intense if student slows down their pace after just a couple months on the system – not only is there a wave of reviews, but they’re not used to doing even the normal amount of review that it takes to maintain a large knowledge profile. At the beginning you get way more lessons than reviews because you don’t have much newly acquired knowledge to maintain. The review ramps up gradually. So when someone starts out really active and then slows down, it compounds that gradual review ramp-up with the wave-of-review effect creating something of a review tsunami.

We still artificially break it up and force a lesson every ~3 reviews on average, even when there’s a big backlog of due review. That reduces the intensity of the review wave while spreading it out over a longer period of time. You still have a big review bill due, but instead of saying “you have to rebalance right now; you have to pay it all off right now before you can make charges again,” we say “you can rebalance gradually; you can continue charging providing you’re making payments equal to 3x what you’re charging.” But that’s about the best we can do. We can allow you to rebalance more gradually, but you’re still going to have to rebalance, and you’re going to be paying off more than you’re charging for a while. It’s the only way you’ll clear your debts.

> Math Academy reviews feel challenging. Aren’t reviews supposed to be easy if I learned the material properly?

If you’re actually trying to maximize learning efficiency, then reviews should feel tough. Why? Because recalling tricky information improves memory, while recalling easy information doesn’t.

That’s the whole idea behind spaced repetition: your memory has to get a bit fuzzy before the next repetition, otherwise the desired effect – slowing the rate of forgetting and remembering longer next time – doesn’t happen (or at least not nearly as much). It’s the act of successfully retrieving fuzzy memory, not clear memory, that extends the memory duration.

If review problems are easy, not actually extending your memory duration, then what’s the point? It’s better to learn something new. A maximum-efficiency teacher will intentionally let your memory fade a bit before review so that the act of refreshing your memory actually deepens your long-term encoding, and they’ll use the extra time to cover more new material.

Reviews should feel as mentally taxing as initial lessons. You’re getting better, but the bar for success also is getting higher. Your brain has to hold the memory for a longer period of time – just like a muscle holding a weight.

The analogy to weightlifting runs deep. In the context of spaced repetition, the way you increase the weight is by waiting longer before retrieving the knowledge again. But you also don’t want to wait too long to retrieve the knowledge, because then you won’t be able to successfully retrieve it. This is just like how in weightlifting, you need to increase the weight to the point where you struggle to lift it, but you are able to overcome the struggle. That’s how you build muscle, and that’s also how you build long-term memory. Spaced repetition = “wait”lifting.

> Does Math Academy’s spaced repetition system provide enough practice? I learned some new information on Math Academy but I am not confident in my ability to retrieve it from memory unassisted.

Math Academy’s spaced repetition system should be sufficient for remembering everything: concepts, procedures, definitions, theorems, formulas, etc. However, it can take several weeks of consistent practice before you really feel confident in retrieving recently learned information. During their first several weeks on Math Academy, it is not uncommon for students feel unsure whether the information they are learning is going to stick in their brain long-term – but after a month or so of consistent usage, they notice something has changed and they’re able to pull a lot of this information from memory without much effort (it feels kind of like magic).

This phenomenon can be explained by the following dynamics within the spaced repetition process:

• Early on, forgetting happens so rapidly that the spaced repetition process is unforgiving to imprecision: if you are slightly late to the next repetition, your memory may have decayed enough since the ideal repetition time that you need a retrieval cue or reminder during the next repetition (and on the flipside, if you are slightly early, then the repetition may lose a lot of its effectiveness in slowing your rate of forgetting).

• However, by the time you get through a handful of repetitions, your rate of forgetting has slowed enough to make the spaced repetition process more robust to imprecision: even if you are slightly late to the repetition, this lateness is small relative to the repetition interval which is now large, so your memory hasn’t decayed much more than desired and you’re still able to recall successfully without reference material. (Likewise, if you are slightly early, the repetition still retains most of its effectiveness.)

Note, however, that even with consistent practice, this “magical transition” depends on properly engaging in retrieval practice, trying your best to recall from memory instead of automatically going back to reference material whenever you feel your memory is a bit fuzzy. Successfully retrieving a fuzzy memory is the very thing that slows future forgetting and extends the memory’s duration. If you just load up the information into your brain by looking at a reference, then you may refresh the information, but you don’t actually slow the rate of forgetting, so you end up stuck in a vicious cycle of constant forgetting and reliance on reference material.

> I know reviews are happening, but sometimes I’m waiting weeks for the first review and months for the second, even when I’m sure those topics aren’t getting any implicit review credit from other tasks I’m doing. I’m not struggling to solve the problems, though sometimes I need to look back at the reference topic for a formula that I’ve forgotten. But shouldn’t the reviews be coming sooner?

When a student demonstrates a high degree of performance on the system, the system adapts to move at a high pace of learning. One component of this adaptation is that the student’s reviews are spread out further over time. Of course, if the student were to start struggling, the pace of learning would slow down and they would receive more frequent reviews.

Keep in mind that the spaced repetition system is not trying to keep a student’s knowledge 100% fresh. It’s trying to minimize the amount of review necessary to keep the forgetting from getting so bad that the student hits a wall in the future and/or has to re-learn the topic from scratch. If a student wanted to get their knowledge up to being 100% fresh on some course (e.g., for an external exam), they would enter test-prep mode and get pummeled with early reviews for a few weeks.

All this said, there is room for us to more precisely calibrate our spaced repetition system in the future, and it’s on our to-do list – though, currently, the spaced repetition system seems to function well enough to serve its purpose, and there are weaker links in the chain that we need to focus our efforts on improving.

> After I complete a course and move to the next course, won’t I forget what I’ve learned in the first course unless I keep on reviewing it?

The spaced review system operates across a student’s entire knowledge profile in general, not just the specific course they’re in. Every topic a student learns on Math Academy will automatically be reviewed into the future, even if they switch to a different course. (Note that these reviews will be implicitly “knocked out” by material in the new course when possible, so students may not see very many reviews that are explicitly on topics from past courses, even though those lower-course topics are indeed continually being reviewed into the future in accordance with the usual spaced repetition procedure.)

> It feels like most of my reviews are on topics I did recently. Shouldn’t most of my reviews be on things that I learned a long time ago?

This is a natural consequence of how spaced repetition works. It’s expected that reviews will tend to be on recent topics more often than older topics. Recent topics need to be reviewed more frequently and there’s less knowledge built upon them (so, less opportunity to find and leverage encompassing to knock the reviews out implicitly). Older topics have already accumulated plenty of review (much of which might be implicit), so they’re not going to need to be reviewed as often, and when review is due, there’s a much greater chance we’ll be able to find encompassings to knock them out implicitly.

Additionally, whenever a due review on an older topic can be knocked out implicitly by a review on a newer topic that is not yet due, we will still serve the newer topic, because doing so not only knocks out the due review on the older topic but also has the added benefit of moving the newer topic a partial repetition forwards along the spaced repetition process. (The repetition on the newer topic is discounted because it is early, but the partial repetition credit still pushes future review off to a later date than was originally scheduled.)

> I know the system intends to review material from older courses, but there are some older topics where I’m not getting any more reviews. Why not?

If it seems like something is not getting reviewed anymore, then one of the following things is happening:

1. It’s getting implicit review credit from more advanced work that you’re doing. For instance, you’ll probably never see an explicit review on ax=b equations because you’re constantly practicing this subskill as you climb up through more advanced math, and this starts happening as early as ax+b=c equations.

2. The spaced repetitions are being massively spaced out. Spaced repetition intervals can grow arbitrarily long, even a year or more (i.e., you might reach a point where you reasonably wait a year or more until the next review). This will happen if you’ve done many reviews on a topic, but it can also happen if the topic is intrinsically easy and you aced it. For any given topic, your spaced repetition schedule is calibrated based on how well you performed on that topic, and how hard the topic is overall (based on its average quiz performance across all students). If there’s a topic that pretty much everyone answers correctly on quizzes, and you ace the lesson, then your repetition 1 interval for this topic might be the same length as, say, a repetition 5 interval for a challenging topic where you just barely passed the lesson and people tend to miss it more often on quizzes in general.

3. This is rare, but it’s technically possible we might have an overly aggressive encompassing set, i.e., the model thinks a subskill is encompassed by a more advanced topic when this isn’t really true. Again, this is very uncommon, but it’s on our to-do list to automatically double-check encompassing weights. However, the spaced repetition system is one of the stronger components of the system at the moment, so this has not been a priority as there are more impactful things to work on.

4. If the topic is in a much lower course, more than just a few courses back, then the “mastery floor” will pave over it and consider the topic completely mastered to the point that you no longer need any practice with it in the future. Right now, having a mastery floor is necessary to prevent the diagnostic algorithm from assessing and potentially over-reacting to a student’s errors in much lower-level material. For instance, if a student is placing into calculus, we wouldn’t want to ask them diagnostic questions on “division using box models” from the 4th grade course – they might know how to do long division just fine but not be familiar with the “box model” approach which is often used to scaffold students into long division. The “quick-and-almost-always-works-fine” solution right now is to limit our collection of missing prerequisites to a few courses back, and assume the student is 100% rock-solid on material lower than that. This works out just fine in the vast majority of cases. It’s on the to-do list to make the diagnostic algorithm more sophisticated to remove the mastery floor, and we’ve figured out how to do this, but it’s a pretty complicated and extensive update under the hood (and again, the spaced repetition system is one of the stronger components of the system at the moment, so it has not been a priority as there are more impactful things to work on).

Remember that the spaced repetition system is not trying to keep your knowledge 100% fresh. It’s trying to minimize the amount of review necessary to keep your forgetting from getting so bad that you hit a wall in the future and/or have to re-learn a topic completely from scratch. (If you wanted to get your knowledge up to being 100% fresh on some course, e.g., for an external exam, you would enter test-prep mode and get pummeled with early reviews for a few weeks.)

The system is intentionally trying to wait for you to get fuzzy on a topic before bringing it back for review, because recalling a fuzzy memory is what really improves retention. Recalling a memory that’s already pretty clear will refresh the memory, but it won’t actually improve retention that much in the sense of slowing your rate of forgetting. Spaced repetition is like weightlifting where the wait is the weight. (Note that retention is also improved by layering on new knowledge that connects to and more deeply ingrains old knowledge – so, when you learn ax+b=c equations immediately after x+a=b and ax=b equations, that’s still extending your retention of x+a=b and ax=b even though you might not be fuzzy on x+a=b and ax=b equations.)

The overall takeaway: while there is room for us to more precisely calibrate our spaced repetition system in the future, don’t worry too much about a small number of topics that you think you’ve forgotten and haven’t been getting reviews on. You’re good as long as you’re able to keep making progress through new material without having to spend a bunch of time re-learning prerequisite material. (If you’re having to quickly peek back at reference material for prerequisites once in a while, that’s fine – that’s very different from spending a bunch of time re-learning prerequisite material.)

Note that, overall, it is uncommon for adult students to experience excessive friction due to forgetting previously learned material on Math Academy. Interestingly, over the years, we have had one-on-one conversations with a handful of students who did experience excessive friction due to forgetting, and in every single instance, the core of the issue turned out not to be the spaced repetition schedule, but rather the student’s behavior when solving review problems. They weren’t actually engaging in retrieval practice. Instead of trying to recall information from memory, they’d default to looking up information from a reference, sometimes even solving problems alongside the reference.

(Remember that spaced repetition doesn’t work if you’re just re-consuming information – you have to be reproducing it from memory in order to really extend your memory duration, peeking back at the reference as sparingly as possible, and then closing the reference and re-pulling the information from memory after peeking. The point of the reference is to, only when absolutely necessary, give you a little help getting over the hump of retrieving information from memory, kind of like a spotter in the gym. The spotter shouldn’t be lifting the weight for you – they should only get involved if you can’t lift it yourself despite trying your best, and even then, they should only provide just enough assistance for you to just barely get the weight up.)

Avoiding proper retrieval practice is more common in kids, and especially adversarial students, but even well-intentioned and responsible adults can sometimes fall into this trap without realizing it. Along these lines, behavioral coaching – making sure learners are engaging in proper retrieval practice, using reference material appropriately, and not falling into any other traps with unproductive micro-behaviors when completing their Math Academy work – is currently going to be more impactful than further calibrating the spaced repetition system.

> I have two reviews on my dashboard and one topic is a prerequisite of the other topic. Shouldn’t the prerequisite review get knocked out implicitly?

There are many cases where a topic requires conceptual understanding of a prerequisite but doesn’t actually provide full practice reps on the prerequisite and consequently will not “knock out” a due review on the prerequisite.

For instance, a review on our 2x2 matrix diagonalization topic wouldn’t knock out a review on our characteristic equation topic because the characteristic equation topic is mostly in the 3x3 setting, which is significantly more challenging due to a more complicated determinant calculation, more complicated factoring techniques for cubics, and sometimes even the rational roots theorem. The review on 2x2 diagonalization may provide a bit of fractional credit towards the characteristic equation, which would push out the next review a bit further, but it’s not enough to knock out a currently due review.

This “non-encompassed prerequisite” situation also tends to show up very frequently in proof-based math. For instance, you might use eigenvalues/eigenvectors in a proof replacing Av with λv, but this does not actually provide practice calculating eigenvectors/eigenvalues.

> Why are there multiple questions in reviews? Why not just one question?

There are several reasons why reviews contain multiple questions:

1. Reviews need to provide interleaved (mixed) practice across multiple knowledge points in the original topic.
2. Reviews need to assess the student’s level of mastery. Mastery means consistently solving problems correctly – not just once.
3. Assessing multiple questions helps make passing the task robust against guessing.

Additionally, providing several review problems feels in line with what a human expert tutor would do.

> I got three questions correct and only two questions incorrect. Shouldn’t I have passed the review?

The order of correct versus incorrect is very significant when it comes to measuring learning. Underlying factors need to be considered too, not just surface-level aggregate counts.

For example, the two scenarios below both entail getting three of five questions correct but have very different interpretations:

• ✖✔✖✔✔ is interpreted that the student was initially a bit confused, but then learning occurred and then they started really “getting” it.

• ✔✔✖✔✖ is interpreted that, despite getting some initial questions correct, the student started struggling afterwards, indicating that they were not really “getting” it initially (at least, not as much as the first two answers would have suggested).

Interleaving

> Why can’t I just learn one unit at a time? Interleaving feels disorienting.

We realize it may feel easier to learn one unit at a time without interleaving. However, that feeling is completely artificial: students measurably learn better when they vary up their practice after a minimum effective dose of initial learning.

As discussed in The Math Academy Way, a common finding in the research is that when students do not interleave, learning tasks are made artificially easy because the surrounding context is already on one’s mind – one does not have to pull the context from memory again. This produces a comfortable sense of fluency, but that feeling is completely artificial.

Think about it this way: maybe you go through a lesson on limits and you’re feeling really good, and then we change things up on you and give you a lesson on derivatives, and then integrals, and then sequences & series. And after that we go back to limits. You might say “You messed up my learning! I had limits down pat and now you made me forget it by making me think about other stuff.” But the thing is, if you can’t retrieve that information from memory instantaneously, after thinking about other things, it means you didn’t really have it down pat. You just felt like you did because you weren’t being made to attempt to regenerate the information from scratch, unassisted, from memory. And the only way to get better at regenerating the information from scratch, unassisted, from memory, is by having to practice doing that – which only happens when you interleave.

> Isn’t blocked practice superior to interleaving when a student is first learning new material?

Yes – which is exactly why we have students engage in blocked practice instead of interleaving when learning new topics. New topics are presented as lessons which consist of blocked practice for a few blocks of gradually increasing difficulty. Material is only interleaved on reviews and quizzes after a baseline level of mastery has been demonstrated via passing the lesson. This transition from blocked practice to interleaving is aligned with the research literature.

> My tasks used to be spread out really nicely along a cross-section of the skill tree, but I’ve recently been getting lots of tasks from the same area of the skill tree. What happened to the interleaving?

Statistically, the system does a solid job of spreading out topics on average. However, if a student is missing a large swath of foundational subskills concentrated in a particular section of the skill tree, that can severely restrict the variety of available learning paths, and the system will have no choice but to hammer on that area of the skill tree in order to open up more learning paths.

The most common culprit is trigonometry: when a student (especially an adult) comes in with sizable but incomplete mathematical foundations, it’s not uncommon for the missing foundations to be concentrated on trigonometry. Trigonometry itself contains lots of topics, is very hierarchical, requires quite a bit of memorization, and is typically not covered comprehensively in school, yet it shows up frequently in precalculus, calculus, and beyond. These factors make it a prime breeding ground for some of the most massive knowledge gaps.

This phenomenon is not unique to mathematics – it happens across skill trees in general. Personally, it actually happened to me while climbing the figure skating skill tree. I used to play hockey, so I came into figure skating really solid on many of the foundational skills, but there was one particular area that I was incredibly weak on: smoothly transitioning between forwards and backwards while balancing on a single leg. This skill area is a subtle prerequisite for pretty much every cool figure skating move you can think of. So, I had to just focus on hammering that subskill area for a while to open up a wider variety of progressions. Of course it would have been nice to have a greater variety from the beginning, but there just wasn’t much else that was productive to work on. But after shoring up that particularly glaring foundational weakness, I opened up a nice variety into the beginning of various progressions into jumps, spins, etc.

That said, it’s on our radar to check for additional optimizations that we can perform to better detect an impending ramp-up of concentrated work even further ahead of time, and try to spread out the density more evenly over time.

Remediation

> If a student struggles on a task, why does the system halt it and come back to it later? Why not continue hammering on that task and force the student to get through it before working on anything else?

As discussed in The Math Academy Way, on the rare occasion that a student fails a lesson, we temporarily pause along that learning path so that some of the initial learning can consolidate into long-term memory for the second attempt (which lightens load on working memory, decreasing the chance of cognitive overload, and increasing the chance of success).

We recognize that some students may prefer to keep hammering and attempt to close the loop right away instead of pausing, 1) it’s not efficient to keep hammering if things aren’t clicking, especially when you consider the opportunity cost of other things you could be learning successfully in that time, and 2) continued hammering typically results in lots of frustration.

It’s much more efficient to take a break, move forward with some wins along other learning paths that don’t depend on said topic, and then come back better set up to succeed the second time around now that some of your initial learning has consolidated and you’ve got some successes under your belt. This approach has worked wonders for us: 95% of lessons are passed on the first try and 99% within two tries. By halting a failed lesson and coming back to it later, that produces an 80% chance of passing the lesson the second time around without any additional intervention.

(On the extremely rare occasion that a student fails the lesson again on the second attempt without getting any further through it, we immediately assign reviews on the “key prerequisite” topics most relevant to the segment where they plateaued, so that the student is completely fresh on the foundational skills most relevant to their point of struggle and can bust through the plateau.)

> If a student fails a task, why does the system typically ask them to re-attempt the same task later? Why doesn’t it just peel back their knowledge profile immediately?

How quickly the system peels back a student’s knowledge profile in response to a failed task depends on how much evidence the student has demonstrated for knowing the prerequisite topics. If a student has demonstrated strong evidence for knowing the prerequisites, then the system will be slow to peel back the student’s knowledge profile: if a student fails a lesson twice in a row without making additional progress, the system will provide support in the form of remedial reviews on the key prerequisite material implicated in the student’s point of struggle, and if the student fails those remedial reviews, then they will be given the corresponding lessons, and so on.

This is a slow process because it has to be resistant to adversarial students “gaming the system.” If we peeled back a student’s knowledge profile quickly in response to failing a task, even when there is strong evidence that a student knows the prerequisite content, then it would create an exploit: whenever tasks begin to feel challenging, an adversarial student could intentionally fail a number of tasks to peel back their knowledge profile until they reach the point where they have days of super easy work ahead of them that they already know how to do.

That said, if a student supplied evidence for knowing material, but the amount of evidence is very low, then the system will be faster to adapt. This process is explained in more detail in the answer to the FAQ entry “After the diagnostic, why did failing a single task bring my progress down multiple percent?”.

> If a student gets a question incorrect, shouldn’t the system try to explain things differently?

Some people think that students need a million different explanations of the same topic until one “clicks” for them. But really, if you have to explain something a ton of different ways to a student before it they can follow that explanation well enough to successfully engage in active problem-solving, then either

1. your original explanations were not good in a pedagogical sense, or

2. the student was lacking prerequisite knowledge and the explanation that “clicked” managed to circumvent that prerequisite knowledge (which often indicates that it’s reducing the topic to a simpler case that doesn’t involve the prerequisite – which means the curriculum is watered down and the student will only be able to solve cherry-picked problems).

When you have

• highly scaffolded, carefully curated content,
• that has been battle-tested over a large number of students,
• and continually analyzed to detect and further scaffold any areas where more than a sliver of students fall off the rails,
• and has gotten to a point that 95% of students pass lessons on the first try (and 99% within two tries),

if you give a lesson to a new student who has mastered all the prerequisite material, then there’s really no excuse for them not to be able to learn it.

For lessons that have undergone this much data-driven refining, on the rare occasion that a student does struggle with it, it doesn’t mean that the lesson needs to explain things in a different way. Usually, all it takes to rebound is a bit of rest and a fresh pair of eyes. And then the same exact content will “click” the next time around.

> If a student passes a lesson but doesn’t get full XP, is extra remediation needed?

We only let a student pass a lesson if they evidence a sufficient level of mastery to continue building on the information they learned, which is the most efficient way to deepen one’s level of mastery. If a student passes a lesson, then no remedial support is needed.

Of course, this “baseline level” mastery is not synonymous with maximum understanding (i.e., “completely intuits everything covered in the lesson to the point of full automaticity”), and different students will be at different levels between baseline mastery and maximum understanding after completing a lesson. However, after establishing baseline mastery, reaching the maximum level of mastery is a more gradual process that happens as the material is continually revisited into the future, showing up on reviews, quizzes, and as a component skill/concept in more advanced material. As the student continues reviewing and layering more advanced knowledge on top of what they learned, their understanding will become further solidified and they will move closer and closer to the point of maximum understanding.

> Does Math Academy work for every student? What about students who are below grade level?

A student does not need to be advanced or even at grade level to be successful with Math Academy. The effectiveness of Math Academy does not depend on a student’s level of knowledge relative to their grade level, but rather on whether the student’s behavior aligns (or can be made to align) with the learning process. Math Academy will work, spectacularly well, for students who are willing to put in a consistent effort.

When a math learner struggles, the root cause of struggle can typically be traced back to instructional leaps and/or knowledge gaps:

• the student may be presented with too much new information at once (an “instructional leap”), or
• the new information may depend on lower-level material they either missed, never fully mastered, or have forgotten (a “knowledge gap”).

Math Academy resolves these pedagogical shortcomings by automatically detecting and filling knowledge gaps and scaffolding new information into bite-sized pieces:

• Every student starts with an adaptive diagnostic that not only identifies their level of knowledge within their course, but also checks for any missing prerequisite knowledge – and if any knowledge gaps are found, they are automatically added to the student’s learning plan so that they can be repaired. In this way, Math Academy creates a custom math course for every individual student.

• In addition to filling existing knowledge gaps, Math Academy also prevents new knowledge gaps from occurring by leveraging mastery learning and spaced review. Students are provided with as much practice as needed to reach mastery, they are only asked to learn new topics for which they have mastered the prerequisites, and they periodically review previously learned material so as not to forget it.

• We make the steps of our “learning staircase” incredibly small and leverage analytics to continually refine our content, adding extra scaffolding where needed to ensure that on every lesson, the vast majority of learners who attempt it reach a sufficient level of baseline mastery on the first try. On the rare occasion that a student fails a lesson twice in the same place, we provide additional practice on the prerequisite material most relevant to the student’s specific area of struggle before asking them to reattempt the lesson.

That said, there are a few caveats to keep in mind:

• If a student is not aligned with the learning process, then simply resolving pedagogical shortcomings will not be enough to produce successful learning. For this reason, we strongly recommend that parents sit with their children in the early stages to ensure they’re studying effectively and aren’t guessing or rushing. Once kids have built up effective study habits, they can work independently – and if they’re serious about learning math, they will make incredible progress.

• While Math Academy does avoid instructional leaps, individual differences in cognitive ability (e.g., working memory capacity) do affect the level of scaffolding required, and there is a level of cognitive disadvantage at which a student may require the assistance of a human math learning specialist to help break things down further.

Task Dashboard

> Sometimes I have some reviews in my task queue, but then I do a lesson or two, and the reviews disappear from the queue. Don’t I need to do them?

This is expected behavior because tasks are selected dynamically. Sometimes a student might have a lot of due reviews, but after a student completes some of those reviews it’s a better use of time to complete some new lessons and make a bit of forward progress before going back to the due reviews. And sometimes making a bit of forward progress will open up new lessons that knock out previous reviews. It’s always a balancing act, we’re always trying to serve tasks that are optimal for the student to work on at this specific moment in time, so the options available are always subject to change. The way to think of the dashboard is not a task queue, but rather an ever-changing menu at a math buffet. The menu is always constructed to try to nourish students in the ways that they’re most in need of at that moment in time.

> Given that lessons can “knock out” reviews, should students always give preference to lessons over reviews if both activity types are available?

It doesn’t really matter. If a review is on a student’s dashboard it means we weren’t able to knock it out by having the student do a lesson instead. Whenever it is possible for a due review to be knocked out by a lesson, we will only offer the student the lesson. We will not offer them a review that is made redundant by a lesson already on their dashboard.

> Is there any logic to the ordering of lessons on the dashboard? Is it better to start with lessons shown at the top?

The short answer is it doesn’t really matter. The long answer is that they’re ordered by “importance,” where importance is a combination of how much review they knock out implicitly, how much further content in the course depends on them, and various other factors. However, all these tasks have a very high importance rating, so we recommend just picking whatever task looks the most appealing.

> Shouldn’t the entire selection of tasks on my dashboard change with every task that I complete?

Once a task is selected, it typically stays on the dashboard. Completing a task will influence future tasks that appear on the dashboard but it typically won’t kick existing tasks off the dashboard. The efficiency loss turns out to be negligible, especially when we’re trying to maintain a diverse spread of tasks on the dashboard anyway. The efficiency loss turns out to be negligible, especially when we’re trying to maintain a diverse spread of tasks on the dashboard anyway, and it’s desirable to keep existing tasks on the dashboard for the following reasons:

1. Selecting new tasks is not instantaneous.

2. Many people will try to avoid tasks indefinitely unless they’re forced to do them,

3. If someone is not wanting to do a task that we’re forcing them to do, it feels more fair and takes the edge off if it’s been sitting around for a while: “I told you on Monday that you had to clean your room this week, I reminded you every day, and now it’s the weekend, so no you can’t go out with your friends until you clean your room, and don’t pretend I’m blindsiding you here, you knew this was coming.”

4. We’ve experienced that lots of people get frustrated when a task is removed that they were hoping to do (which may seem like a silly issue in theory, but in practice it’s human nature to get frustrated when something you were looking forward to is taken away).

STUDENT BEHAVIOR

Usage of Paper and Pencil

> Should Math Academy students take notes during lessons?

Note-taking should not be necessary. Our spaced repetition system takes care of review, and on the rare occasion that you do happen to forget something, you can always look back at prerequisite lessons in “reference mode” to brush up on whatever you may have forgotten.

Even further, we would actively recommend against taking notes. To transfer information into long-term memory, you need to practice retrieving it without assistance – but when you take great notes, you’re tempted to refer back to those notes all the time instead of trying to pull information from memory. As a result, notes can turn into a crutch that spirals you into a vicious cycle of forgetting. The same reasoning applies to any sort of reference material.

Of course, if you can’t recall something after trying your hardest, it’s okay to check reference material, but only as a last resort. Even then, do not solve the problem alongside the reference material – peek once, and then try to solve the problem without looking again.

That said, we wish to make a distinction between “note-taking” and “listening on paper.” While we do not recommend transcribing information for later use, we see no issue with jotting down key bits of information to maintain focus and draw connections while being presented with new material. “Listening on paper” is not necessary or even helpful for all students, but some find that it helps them engage in “active listening” and deepen their processing of the material being learned.

Again, however, a student who practices “listening on paper” must always take care to avoid the following pitfalls:

• Pitfall 1: Transcribing, or, more generally, slowing down the rate of ingesting new information without actually deepening the processing of that information.

• Pitfall 2: Solving problems alongside notes, or, more generally, using a reference as a crutch to avoid or reduce effort towards proper retrieval practice.

> When should a student solve a problem in their head vs writing it out on paper?

We would suggest that the most technically correct general rule is this: Do a problem (or sub-component of that problem) in your head when

1. you’re able to do it in your head reliably, with very little effort, and

2. it feels like you’re only holding one thing in your head – like, you’re working with a solid, cohesive “chunk” of information as opposed to having to “juggle” multiple components of that information to keep it in your brain.

Here’s the science behind that.

It’s well established in cognitive science, specifically under the umbrella of “cognitive load theory,” that your working memory has a limited capacity to hold new information – and when you push your working memory close to that limit, you become more likely to make mistakes and less likely to complete the training task, which impedes your learning.

Reducing cognitive load is the goal – not just “a” goal, but in fact “the” goal, the whole point – of structured education. The more instructional scaffolding is provided, the lower the student’s cognitive load, and the less cognitive “friction” there is to slow the student’s acquisition of the knowledge covered in the curriculum.

It’s the same way with solving problems on paper: the goal is to lower your cognitive load. By writing down intermediate steps on paper, you can temporarily remove information from your working memory to make room for new information, and then quickly load up the original information by looking back at the paper when needed.

(In a sense, the paper functions as an artificial long-term memory bank where you can store new information that you don’t already have encoded in your brain’s long-term memory. It takes a lot of time and effort for your brain to encode information to its own long term memory, but writing information on paper enables you to sidestep these biological limits.)

The Asymmetric Tradeoff

To be clear, there is a hidden tradeoff. If you overdo the scaffolding, or you write down more than you need to on paper, then it’s going to inflate the amount of time that it takes you to complete the curriculum or solve the problem, respectively. If your cognitive load is already low, there is no benefit to lowering it further – all that does is create more mechanical work for you that burns your time.

However, the tradeoff is asymmetric:

• If you undershoot the scaffolding or don’t write down enough work on paper, and you blow your working memory capacity, then you hit a brick wall. You’re simply unable to complete the task. Your learning progress grinds to a halt, full-stop. Even if you just “come close” to full capacity, your error rate skyrockets, impeding your learning.

• On the other hand, if you overshoot the scaffolding or write down more than you needed to on paper, then sure, it will technically be suboptimal, but typically not by much. You wrote down an extra line or two on paper than you really needed to? Big whoop, it took you an extra couple seconds to solve the problem. You could have saved a couple seconds by not writing those steps down, but that would also have put you dangerously close to holding too much in your head and making a mistake. Just like in your bank account, having a little buffer is not a bad thing.

Because the tradeoff is so asymmetric, it’s best to err on the side of caution, writing down potentially a bit more than you need to. When in doubt, write it out.

Building Good Habits

In addition to guarding yourself against being on the wrong side of the asymmetric tradeoff, another reason why you should err on the side of caution (i.e., writing down too much as opposed to too little) is that you need to build good habits for the future.

As you climb up the levels of mathematics, the level of technical sophistication increases, and consequently, so does the level of cognitive effort. Even if you are able to do problems entirely in your head at lower levels of math, you will not be able to do so indefinitely into the future. You will eventually get to a point where you are unable to do problems in your head without blowing your working memory capacity, and at that point, the only way to continue making progress will be to use paper and pencil as a tool to reduce your cognitive load.

However, the longer you go without using paper and pencil, the more solidified that habit will be, and the harder it will be to get yourself to change it. So, even if it’s not strictly necessary, it’s a good idea to get in the habit of writing at least some work out. If you don’t, then you may cling to the habit of doing all the work in your head for too long, well past the point when you really need to start writing work down on paper – which will gradually eat away at your performance and progress, eventually bringing you face to face with a “day of reckoning” where your entire mathematical future is on the line.

We have seen many bright students breeze through basic math refusing to write down any work, only to struggle in intermediate or advanced math simply because they stubbornly continue refusing to write down their work.

Reliance on Reference Material

> When should I look back at reference material?

To learn effectively, balance self-reliance with use of resources:

1. Always begin a problem by attempting it independently. Do not immediately refer back to reference material.

2. If genuinely stuck, peek at related examples or explanations to remind yourself of the next step in the problem-solving process.

3. After a quick peek, return to solving the problem on your own to apply and reinforce what you learned.

Strive for independent problem-solving as much as possible. Use reference material when necessary, but only when necessary. Do not allow it to become a crutch.

> During a quiz, if I can’t remember a “fact” (e.g., a definition or theorem) but I remember the process for using it to solve problems, should I look it up?

We would recommend not to rely on any external material during the quiz, as it will cause the system to make decisions as if you got the question right without relying on any reference material. In the case described, you would benefit from getting a refresher on those topics via follow-up reviews after the quiz – but you won’t get that support if the system thinks you were able to answer the question without looking at a reference.

That said, we would recommend taking your best guess even if you are not confident about your answer. If you’re fuzzy on information, then it’s possible to retrieve the information successfully despite having a low degree of confidence. (That said: even if you manage to get it right, definitely look at the solution afterwards to further refresh your memory.)

More generally, we (people in general) are often not good at judging how well we know something, especially if it’s something that we learned recently and are not super confident about. The Math Academy system has been designed around the idea that the student should just take their best attempt at whatever they’re doing, and the system will take corrective action after any attempt that turns out to be unsuccessful.

> What should a student do if they are unsure how to solve a problem despite trying their best to refer back to the supporting instructional content?

This situation should almost never happen if a student reads the supporting instructional content carefully and attempts to solve the problem step-by-step, writing every step down on paper, and referring back to the worked example at each step when stuck.

However, if this situation does still arise, and the student spends 5 minutes stuck at a particular step without making further headway on it (despite reviewing the key prerequisites and checking earlier parts of the lesson for information they may have missed), then the most productive use of time is to submit a “best guess” and then study the solution carefully after the question is graded (regardless of whether the best guess was correct or incorrect). The student should not move forward to the next question until they have worked out the original question themselves, on paper, following along with the solution and ensuring that they understand the rationale behind each step.

If a student ever finds something that could have been explained better in the supporting instructional content, they are encouraged to submit a flag explaining the improvements that they think should be made. We continually refine our content based on student feedback, and while the content is very solid by now after years of refinement, we are always on the lookout for ways to continue improving it.

Adult Guidance

> As an adult, is there anything I need to do to ensure my child succeeds on the system?

Sitting With Your Student

For younger students starting with Math Academy, adult supervision is crucial initially. We recommend a parent or guardian sits with the student for at least the first few lessons to

• help navigate the system effectively,
• model good learning behaviors and strategies, and
• ensure proper engagement with material.

During these sessions, the adult should demonstrate

• careful reading of instructions,
• systematic problem-solving, and
• effective use of reference material (worked examples, solutions to previous problems, prerequisite lessons).

This early guidance builds a foundation for successful independent learning.

Developing effective habits takes time. Many students need days or weeks to fully grasp and independently implement productive engagement with Math Academy tasks.

Stepping Back

While it is necessary to sit with the student initially, the goal is to gradually step back and empower the student to thrive as a fully independent learner. This transition must be guided carefully:

1. Start by closely monitoring the student’s behavior and thought processes during their learning tasks.

2. Once the student has demonstrated the ability to successfully and consistently complete tasks without intervention, allow them to complete individual tasks unsupervised. Have the student report back with every task they complete.

3. Progressively decrease the frequency of check-ins. Once a student has demonstrated successful independent learning while checking in after every task, move to daily check-ins, then every-other-day check-ins, then every-few-days check-ins, and finally weekly check-ins.

4. We recommend checking in at least once every week, even if the student is succeeding independently.

Don’t assume self-sufficiency too early. Monitor progress and be ready to step in if needed. Some students may require extended guidance, which is normal.

Remember, the goal is to foster independence through a scaffolded progression. As students develop good habits, they can transition to more independent work, preparing them not only for Math Academy but also for future academic challenges.

Incentive Structures

A structured incentive system plays a critical role in helping younger students stay engaged with their learning. While it is important to guide students through their early sessions, equally vital is establishing a reward and accountability system that motivates them to stay on track independently.

Without clear incentives and regular accountability, students may fall into the trap of minimal effort. For instance, they might

• skim through instructions instead of reading them thoroughly,

• rush through practice questions without reflecting on worked examples, and

• appear engaged while being distracted by other activities like browsing unrelated websites or daydreaming.

To prevent this, parents or guardians must establish an accountability system:

• Sit with the student and ensure they are making genuine progress, not just going through the motions.

• Simple incentives can be powerful. For example, “Complete all your lessons this week, and we’ll get ice cream on the weekend” or “No video games until today’s tasks are finished.” Tailor these rewards to something the student genuinely values.

• Even when a student becomes more independent, daily or weekly check-ins are necessary to maintain progress. If a student’s effort diminishes, sit with them again to refocus their engagement.

This combination of incentives and consistent accountability ensures that students develop both the discipline and independence needed for success.

> What should my child be doing at each step of the learning process?

1. Prepare to Learn. Have a pencil and paper ready, find a quiet space for concentration, and ensure basic needs are met (not too tired or hungry).

2. Read for Comprehension. Read carefully and intentionally. Don’t race. After each problem or explanation, pause and ask: “Do I truly understand what I just read?” Re-read when necessary. Younger students in particular, especially those who have recently learned to read, may need guidance in making this transition into “reading to learn.”

3. Engage with Worked Examples. Unless it’s a simple recognition problem, use paper and pencil to work through each step alongside the example. The point of the worked example is to engage in a hands-on walkthrough of the problem-solving process.

4. Slow Down the Thought Process. Don’t rush! As you follow along with a worked example or work through a problem, pause to justify each step that you carry out. Use the worked example as a reference, but do not copy blindly. The goal is to understand and practice, not to transcribe.

5. Approach Questions Diligently. Maintain a careful, intentional approach when answering questions. Unless it’s a simple recognition problem, work out your solution on paper. Resist guessing, even if you think you know the answer. Peek back at the worked example if needed, but try to retrieve information from memory whenever possible.

6. Handle Mistakes. Mistakes are valuable learning opportunities. When you get a question wrong, read the explanation carefully and identify where and why you made the mistake. Rework the problem using the correct method, ensuring you understand each step. This process of error analysis and correction leads to deeper learning.

> Once I step away, how do I know if my student is continuing to use the system properly?

Here are some warning signs that a student may not be using the system properly:

• Failing more than 1 out of every 10 lessons on average. On average, students pass lessons on the first try 95% of the time and within two tries 99% of the time.

• Receiving negative XP. XP penalties are triggered in response to patterns indicating lack of effort (e.g., rushing and guessing). Students who engage in productive learning behavior rarely, if ever, receive negative XP.

• Consistently earning half or fewer XP per minute worked during study sessions (e.g., receiving 15 XP in a 60-minute session). While the rate of earning XP can vary across students and tasks, students who work carefully, with full focus, yet also with a sense of urgency, can typically earn somewhere in the ballpark of 1 XP per minute.

If any of these warning signs appear, a student should check for bottlenecks in their learning process. For instance:

• Are they making silly mistakes due to rushing or trying to do too much in their head?

• Are they getting stuck re-reading about minor areas of uncertainty over and over again, when it would be more productive to move on to problem-solving? (It’s good to re-read when something doesn’t make sense, but if a student is re-reading over and over again without improving their understanding, then working through a problem in a slightly different context can often help clear things up.)

• Are they spending too much time checking and re-checking their answer to minimize the risk of getting it wrong? (It’s good to work carefully and quickly double-check when possible, but it’s also possible to take this to an unproductive extreme where it seriously slows down the learning process.)

There are many possible bottlenecks that may occur – too many to list exhaustively. However, in general, it is usually possible for a student to increase their pass rate and XP rate and avoid negative XP by improving their learning process.

Younger students typically require the assistance of an adult when identifying and resolving bottlenecks in their learning process.

XP AND PRACTICE SCHEDULES

XP System

> If I pass a lesson but don’t get full XP, does that mean I only understood part of the material? If so, how does Math Academy fill in the rest of my understanding?

Math Academy only allows a student to pass a lesson if they evidence sufficient mastery to continue building on the knowledge covered during the lesson. So, if a student passes a lesson, then no remedial support is needed.

Of course, mastery is not synonymous with maximum understanding (i.e., “completely intuits everything covered in the lesson to the point of full automaticity”), and students will be at varying degrees between mastery and maximum understanding after completing a lesson. However, as students continue reviewing and layering more advanced knowledge on top of that topic, their understanding will become further solidified and they will move closer and closer to the point of maximum understanding.

> I can do way more than 1 XP per minute!

The XP is calibrated to 1 XP = 1 minute for an average Math Academy student at that level of math. A student may be able to move significantly faster if they are particularly mathematically inclined relative to other students at that level, especially if they are already halfway familiar with some of the material.

> As I progress through Math Academy’s curriculum, it is gradually taking me longer to earn XP. Why does this happen, and can anything be done about it?

As a student progresses into more advanced content, the amount of time it takes to earn XP will increase gradually, even though the average is staying at 1 XP = 1 minute for students at that level of math.

This happens because, as the math gets more advanced, students who take longer to earn XP are more likely to drop out, which effectively raises the bar for being an “average” student at the next level of math. In general, the further you go in any skill domain, the higher levels you reach, the more talented the other people at that level, and the harder you have to work to get to the next level.

That said, if a student is taking a long time to earn XP, then it is always worth checking for an unnecessary time sink that forms a bottleneck in their learning process.

• For instance, a student might skip over the lesson and then spend a long time getting through each problem because they didn’t read carefully and are trying to solve the problem without relying on any scaffolding.

• Alternatively, a student may spend an excessively long time studying a worked example, hung up on a minor detail or phrase that they feel they are not fully confident in understanding, when it would be more productive to move on to active problem-solving. (Often, actively working through a problem in a slightly different context can clear up minor confusions that may arise when passively viewing a worked example.)

Another possibility:

• A student might make a lot of silly mistakes due to rushing and doing all the work in their head, causing them to have to do many more problems than if they just worked problems out more carefully and accurately on paper.

• On the flipside, a student might spend too much time unnecessarily double and triple-checking against the worked example to make sure they solved the problem correctly. (While it’s good to be diligent and work problems out carefully, it’s also possible for a student to go overboard and move too slowly because they spend too much time minimizing the risk of getting something wrong.)

Yet another possible cause of taking a long time to earn XP is when a student over-relies on reference material, solving problems alongside worked examples. After a student has read the worked example and moved on to solving problems, the student should refer back to the worked example only as a last resort when they have tried their hardest and failed to remember the next step in the problem-solving process. And even then, whenever a student refers back to the worked example, they should only peek at the part they’re stuck on before trying to solve the rest of the problem unassisted.

There are many possible bottlenecks that may occur – too many to list exhaustively. However, in general, it is often possible to increase one’s XP per unit time by tracking where the big time sinks are and trying out strategies to speed up those parts of the learning pipeline.

> I don’t think XP is a perfect measurement of effort.

The “1 XP = 1 minute of focused effort” metric is an average: when we take a large number of serious students and a large number of tasks and compute the XP per time, it comes out to about 1 XP per minute. While we do our best to assign each learning task an XP that accurately represents the amount of work needed to complete it, the observed XP-to-time ratio may vary for any particular student doing any particular task.

Additionally, we recognize that some components of effort are not currently taken into account when assigning XP, and it is on our roadmap to make XP grading even more granular to incentivize those components of effort. For instance, one thing that we’d like to factor into the XP grading is how often a student refers back to the worked example. Getting questions right with minimal reliance on the worked example is more effortful and will move the needle on a student’s learning more than if they refer back to the worked example all the time, by default, instead of trying their best to pull information from memory. This would be part of a larger push on “in-task coaching,” that is, encouraging students to engage in micro-behaviors that enhance learning but aren’t yet incentivized through our XP system.

> Why did I get a penalty? When I make mistakes, I thought the system was supposed to remediate, not penalize!

As discussed in The Math Academy Way, if you fail a lesson twice in a row at the same worked example, then you’ll automatically be assigned reviews on the key prerequisites associated with the worked example. (Though, usually all it takes to rebound is a bit of rest and a fresh pair of eyes.)

As discussed in The Math Academy Way, when a penalty is assigned, it’s usually because the system detected behavior indicative of rushing when careful work is needed. We know it’s fun to fly through questions swiftly when you’re able, but whenever you hit a rough spot, make sure to take the time to work carefully. Don’t sacrifice accuracy for speed.

> I feel like a particular answer choice letter has been coming up more frequently than the others! What’s going on?

We have a validation tool that we run against our database to ensure that correct answers are distributed randomly across the choices. Once in a while, we receive a message from a student who thinks that they have detected a trend, but these messages tend to disagree on the trend (some students may think that “A” is more frequent, others “B”, others “C”, etc.), so it seems unlikely there has ever been a trend at all. Sometimes it does happen even with proper randomization that a short-term trend may appear – kind of like, if you flip a coin enough times, you’ll eventually get a string of arbitrarily many heads in a row.

Practice Schedules

> If I have a limited amount of time to devote to Math Academy each week, should I allocate that time into longer, less-frequent sessions or shorter, more-frequent sessions?

When learning math, it’s best for study sessions to be short and frequent (as opposed to long and sparse). For instance, suppose you’re budgeting 3 hours per week to learn math. It would be better to study 30 minutes six days per week, as opposed to 90 minutes twice a week. There are a handful of reasons why.

1. You want to form a habit. The more consistently you study math, the more it will become a habit that you naturally do each day without thinking, just like (hopefully!) taking a shower and brushing your teeth.

2. You want to operate at peak productivity during your session. During a short 30-minute session, it’s easy to maintain a high level of focus and intensity – whereas, during the second half of a long 90-minute session, fatigue will set in and make you significantly less productive.

3. You want to minimize the amount you forget between sessions. When you have multi-day gaps between study sessions, you’ll have to spend more time revisiting previously covered material.‌ (Just ask any teacher how much their students forget over weekends, and how much valuable class time they have to spend on Monday re-teaching the things that they covered on Thursday and Friday.)

However, there are some caveats to consider.‌

• Whenever you switch to a different activity, it takes a few minutes for your brain to catch up and enter a state of flow in the new context. This is called “context switching cost,” and if you make your sessions too short (less than 20 minutes or so), then the proportion of study time that is wasted on context switching will outweigh the other benefits of daily practice. Consequently, it’s best to spread out your practice as much as possible subject to the constraint that each session is sufficiently long for the context-switching cost to be proportionally negligible.

• Additionally, if you have a hectic schedule and “six days per week” in theory ends up being just “three days per week” in practice, then you’ll need longer sessions just to achieve the same volume of practice.

> If a student completes a Math Academy course very rapidly, working several hours per day for several weeks, will they still learn the material properly?

Yes, the student will still properly learn and master the material. The only catch is that if they were to flat-out stop doing math afterwards, they would forget the material sooner than if they spread it out over a longer period of time (this is simply a consequence of the spacing effect and the mechanics of spaced repetition, discussed in The Math Academy Way). However, if the student continues working on Math Academy afterwards, continuing to learn more math, then they will layer on top of their knowledge and receive any additional reviews that are necessary to maintain their knowledge, so they won’t forget what they’ve learned.

> What’s a reasonable XP pace for a typical Math Academy student?

Think of it like exercise. If you want to level up your abilities, then you should probably aim to get at least half an hour of exercise every other day. And if you’re really serious about it, then you’d probably shoot for a 40-minute workout most days per week. (Note that 40 minutes every weekday will have you moving nearly twice as fast as a half hour every other day, since 40 × 5 is about twice of 30 × 3.5.)

> Is there a maximum daily study duration after which one risks causing more harm than good?

There isn’t a hard limit. More volume equals more progress provided that you’re working productively and not burning yourself out.

• If you’re tired and your head is spinning and you’re making tons of silly mistakes, then it’s time to stop.

• If you’re so fatigued that you can’t help but zone out (or get distracted scrolling through memes) between questions, then it’s time to stop.

• If you skip the next couple days because you’re so blown out from the previous study session, then it’s time to reduce the single-session duration and increase the consistency.

But until you hit those issues, doing more will have you truly learning more and making faster progress towards your long-term goals.

Basically, challenge yourself to put up some serious volume, but also be honest with yourself about whether you are working productively and showing up consistently, and don’t lose the long game trying to win the short game.

> Can Math Academy be used for very casual learning, an hour or two per month?

Math Academy focuses on students who are trying to acquire math skills to the highest degree possible. We teach math as if we were training a professional athlete or musician. We maximize learning efficiency in the sense that we minimize the amount of work required to learn math to the fullest extent. Learning math to the fullest extent requires a dedicated effort of at least a couple hours per week.

We realize that there are many learners who only want to devote an hour or two per month, but, at least right now, such learners would be better served elsewhere. It’s a totally different optimization problem – maximize surface-level coverage subject to some fixed, miniscule amount of work – and as a result it would require a different curriculum and possibly different training techniques (or at least, differently calibrated techniques).

> When kids quickly cover lots of material and move on to advanced math well above their grade level, doesn’t that create knowledge gaps?

We reduce educational friction and empower each student to move at their own pace. For many students, their own pace is surprisingly fast.

As discussed in The Math Academy Way, the main finding from studies into academic acceleration is this: whether a student is ready for advanced mathematics depends solely on whether they have mastered the prerequisites. We leverage mastery learning, meaning that students do not move on to more advanced content until they are able to consistently solve problems correctly in the prerequisite topics.

We’re not pushing students so fast as to create knowledge gaps. In fact, we’re doing the exact opposite. We’re detecting and repairing existing knowledge gaps, and making sure no new knowledge gaps are formed. That’s part of the reason why students learn so efficiently on our system – students are always working on material that is at just the right level for them, as opposed to wasting time struggling to learn things that depend on missing prerequisite knowledge.

We’re not skipping content and we’re not lowering the bar for mastery. What we’re doing is making sure that student learning time is being used as efficiently as possible. It’s not that students on our system are learning shockingly fast, it’s that students elsewhere typically learn shockingly slowly due to unfavorable learning conditions, working on tasks that are too easy or too hard, poorly calibrated to their level of skill. We provide a more favorable learning environment that removes a lot of educational friction.

> I would love for my kid to accelerate, but doesn’t the time commitment have a large opportunity cost in terms of other development activities they could be doing in their childhood?

We’re not asking students to devote an absurd amount of time to studying math. The students in our school program were only expected to do 40 XP/day, equivalent to about 40 minutes of fully-focused work. That was enough to get them through Prealgebra, Algebra 1, Geometry, Algebra 2, Precalculus, and AP Calculus BC in 3 years (6th-8th grade).

I (Justin) taught there for 3 years and I gave them the choice, they could either focus 100% in class and knock it all out, or they could socialize while getting at least half of their work done. Most students opted to socialize while knocking out 20-ish XP during class and then doing the remaining 20 XP outside of class. And that was fine by me. Some students chose to do more work than was expected because they wanted to fly. And that was also fine by me.

> Why did my estimated completion date change?

The completion date is calculated by first estimating how many XP remain in the course based on your recent performance. (The amount of XP remaining depends on the pace of learning, which adapts to your performance – e.g., if your accuracy decreases, reviews will come more frequently and the amount of XP in the course will increase.) Then, we estimate your recent XP/day pace, and finally divide XP remaining by XP/day.

Any fluctuations in performance or pace will affect the completion date, especially at the beginning when you first start out, because the system has to take a “best guess” and then gradually refine the estimates as you build up more history on the system.

Likewise, if you lose credit for any “conditionally completed” topics, that would also push the completion date back. Topics are “conditionally completed” if you just barely received credit for them based on the diagnostic. Retaining this credit is conditional on maintaining a high level of performance on these topics, since the system will adapt more quickly to your performance in these areas of low confidence as you complete more learning tasks. Missing questions on these topics (or their prerequisites) can lead the system to prune back your knowledge profile in those areas to provide more practice.

> I took a break from Math Academy for a few months, and I worry that my tasks may be too hard when I come back.

If all you need is a brief refresher, then you can trigger a diagnostic to be averaged into your existing knowledge profile (or even just try to pick up directly where you left off and spend a bit more time looking back at prerequisites to refresh). However, if your tasks feel overwhelmingly difficult due to forgetting, then you can trigger a “clean slate” through your settings, which wipes your existing knowledge profile and has you take a diagnostic to fully recalibrate it.

Be sure to set a reasonable daily XP goal that you can maintain into the future, stick to it consistently, and protect your habit to the best of your ability. Even on days when life gets in the way, if you do a little bit of math, that will go a long way in terms of protecting the habit, even if it’s not your full daily goal.

When someone gets derailed from their journey to get better at writing, math, coding, an instrument, a sport, or whatever it may be, it’s almost always the same story: at some point they fell off the wagon entirely and never managed to get back on. When the time comes to get back into the swing of things, it’s much easier to speed up a slow wagon that you’re on, than to get back on a wagon that you’ve completely fallen off of. Every time you fall off the wagon, there’s a chance you might not get back on.

DIAGNOSTICS AND CURRICULUM

Diagnostics

> Why is my placement percentage in the course so different from the percentage of questions I got right on the diagnostic?

The diagnostic does not randomly sample questions from the course, so we would not expect your diagnostic accuracy to match your placement percentage. Instead, it adapts to maximize information gain, selecting the questions on which it is most unsure about your knowledge. Consequently, you would expect to get around half of the diagnostic questions correct regardless of your knowledge of the course (unless you are at an extreme end of knowing close to 100% or 0% of the course).

> Is there anything that I should keep in mind while taking the diagnostic to ensure that I get an accurate placement?

This diagnostic assesses your problem-solving speed and comfort. Be honest about your abilities to ensure an accurate, personalized learning path. The aim is finding your ideal starting point, not achieving a high score.

Do:

• Complete within 1-2 days. (It’s fine to take breaks.)
• Take when well-rested and focused.
• Skip questions you can’t solve quickly (3 minutes max). If you can’t solve the question quickly and comfortably, click “I don’t know.”

Don’t:

• Drag out over many days.
• Take when tired. (This leads to mistakes.)
• Guess – not even if you can narrow down answer choices. (A successful guess may cause you to be placed too far.)
• Attempt questions that are beyond your level of comfort. (Lengthy answer times, even if correct, will receive minimal credit and inflate the length of the diagnostic.)
• Use any external resources.

Remember, we’re trying to determine what you can do comfortably and confidently, not what you can barely manage with extended effort.

Young students may require parental supervision to ensure they are following the guidelines above. However, the parent should not provide any information on how to solve problems or even comment on the correctness of answers.

Don’t worry if your diagnostic performance differs from expectations. Even students with good grades may have knowledge gaps, as traditional courses often lack comprehensiveness and allow students to get by without truly mastering the material.

Any knowledge gaps identified by the diagnostic will be automatically incorporated into your personalized learning plan, ensuring that you build a solid foundation for your mathematical journey.

> There were way more questions than I expected on the diagnostic!

As discussed in The Math Academy Way, diagnostics require a larger-than-expected number of questions because

1. our curriculum is hyper-scaffolded, and
2. we assess students not only on the course content, but also on any lower-level foundations they might be missing.

For higher-level math courses, diagnostics may need to assess your knowledge of over 1,000 math topics! Even if it feels like there are many questions on the diagnostic, each individual question provides decisive information about your knowledge of about 10 different topics on average.

Think of it like going to a serious gym where you start with a body composition analysis and strength/flexibility tests at every muscle/joint in your body. In order to get you progressing towards your mathematical goals as efficiently as possible, we need to figure out exactly where your strengths and weaknesses are so that we can perfectly calibrate your workouts to your personal needs.

Of course, if you just want to go to the gym on weekends and walk around the track a few times, then this is probably not a great fit! But if you want to come for a serious workout most days each week and reach a serious level of fitness as quickly as possible, then this is the way to do it.

> Can’t you improve the diagnostic algorithm to cut down on the number of questions?

We have spent a lot of time optimizing our diagnostic algorithm to be as quick and efficient as possible. There are some hard limits in the physics of how small we can reduce the number of diagnostic questions subject to additional constraints on the precision and robustness of the overall conclusions drawn from that information. Our diagnostics are optimized to the point that if we forcibly cut the number of questions in half, your placement wouldn’t match up well enough with your true knowledge frontier, and you’d get frustrated doing tasks that are too easy (or worse, too hard).

Yes, people quit if the diagnostic is too long, but they also quit if they’re not placed accurately. Math Academy’s approach to that tradeoff caters to students who are serious about putting in a large amount of work to learn an even larger amount of math. For such users, an hour or two on the diagnostic is proportionally negligible compared to the amount of work that they plan to commit to learning math. Such users typically find it worthwhile to spend a proportionally tiny amount of extra time at the beginning to ensure a smooth mathematical journey indefinitely into the future.

> I am being served lower-grade lessons that feel irrelevant to my course, and it’s taking too long to make progress in my course. What can I do to fix this?

The only time you’d be assigned a lower-course topic is when it’s a prerequisite of a topic in the course that you’re taking and you got it (or one of its prerequisites) incorrect on the diagnostic. In order to place out of these topics, you would need to provide evidence of being able to solve them (or post-requisite topics) on the diagnostic.

Sometimes, students think that topics are irrelevant to their current course when in fact they are necessary prerequisites. For example, integration might not seem relevant to linear algebra but it’s actually necessary to solve problems in inner product spaces. Likewise, the rational roots theorem, synthetic division, and polynomial factoring might not seem relevant but it’s actually necessary to compute eigenvalues of 3x3 matrices.

If you think you could have done better on the diagnostic, it would be worth retaking it very carefully. Keep in mind that all the system’s decisions are based on your demonstrated ability to solve problems, and it is not uncommon for students to take courses elsewhere yet still not have mastered the content well enough to solve problems correctly, consistently, and in a timely manner.

> I missed some questions on the diagnostic, but I know those topics, I swear! Can’t you just give me credit for them?

You are welcome to take another diagnostic and attempt to demonstrate your knowledge by solving problems correctly! Math Academy is a mastery learning system, so the only way to receive credit for topics is to provide evidence of mastery, i.e., to demonstrate your ability to solve the problems correctly. If you miss questions on the diagnostic, the system is going to infer that you don’t know those topics (and any topics that depend on them), regardless of how well you think you know them.

> There is a topic that I know how to do, but the diagnostic didn’t ask me about it and I didn’t get credit for it.

The diagnostic is fully comprehensive; it continues asking questions until it has evidence of knowledge (or lack of knowledge) for every single topic in the student’s course and foundations. Whatever topics the student is not given credit for, it’s because the student submitted incorrect answers on those topics or their prerequisites. While it is sometimes possible to solve questions from a topic despite not fully grasping a prerequisite, this indicates the presence of “holes” in the student’s mathematical knowledge, and the diagnostic intentionally places students at the bottom of their lowest knowledge holes so that these holes can be filled in.

Placing students at the bottom of their lowest knowledge holes is absolutely critical to ensure student success. If the diagnostic did the opposite, placing students at the top of their highest knowledge holes, then students might initially feel like they are closer to their goals as a result of receiving more credit, but these knowledge holes would sooner or later (and likely sooner) derail the student by causing them to become “stuck” while learning new topics that make deeper use of the prerequisite knowledge.

That said, it is not uncommon for adult students to be extremely rusty on their math while taking the initial diagnostic, and then have an outsized portion of their memory come rushing back afterwards as they complete learning tasks. When this happens, it is sometimes possible for a student to place significantly further by retaking the diagnostic.

Additionally, we are working on a button where students can say “I already know this” on any lesson that they receive and evidence their knowledge by answering a couple advanced questions on the topic. That way, it will be fast and easy for a student to continue fine-tuning their knowledge profile after the diagnostic.

> There’s a question on the diagnostic that I might know how to solve, but I want to re-learn it from scratch. I don’t want to receive credit for it. What should I do?

In general, if you answer a question correctly in a diagnostic, you will likely get credit for it. If you are not comfortable with the question and do not want credit, then what you should do is click the “I don’t know” button.

> After the diagnostic, what if I am asked to complete a lesson for which I have not learned a prerequisite?

Math Academy’s diagnostic exam is highly accurate, but not necessarily 100% perfect, as guaranteeing a 100% perfect placement would require an infeasibly large number of diagnostic questions to be answered. To drastically cut down on the number of questions, our diagnostic exam leverages some loose forms of inference that – rarely, but occasionally – may place a student slightly behind or ahead (but more likely behind) their true knowledge frontier along some learning path.

(For instance, if a student answers a question correctly on a “leaf topic” in some module, then we treat that question as a “representative” for the module and award some credit to other leaf topics in the same module. Otherwise the diagnostic would have to explicitly assess every single leaf topic, which would make the number of questions blow up. The idea is that if a student knows a maximally advanced technique within some cohesive group of topics, then they probably know any other advanced techniques within that group, or they should have enough prior knowledge to brush up on the fly.)

In practice, on the rare occasion that a student’s level of knowledge is overestimated and they receive a lesson for which they have not learned a prerequisite, the degree of overestimation is small enough that the student is able to learn the prerequisite by clicking on the prerequisite and viewing its lesson in reference mode. To the best of our knowledge, there has never been an instance where a student was unable to do so, provided that they took the diagnostic properly and submitted answers reflective of their true knowledge. (In the small number of instances where a student was placed too far beyond their true knowledge to make progress on the system, it has always turned out that the student used an external resource for help during the diagnostic.)

Additionally, even on the rare occasion that a student may have to learn a prerequisite by viewing its lesson in reference mode, this issue will quickly disappear as the student completes more work on the system: the student will quickly reach a point where, for all their available lessons, they have explicitly completed lessons on all the prerequisites.

> After the diagnostic, why did failing a single task bring my progress down multiple percent?

After the diagnostic, there may be pockets of math where the system infers student knowledge but has low confidence in that inference. As the student completes learning tasks, the system will adapt more quickly to the student’s performance in these areas of low confidence. As discussed in The Math Academy Way, these topics are called “conditionally completed” because while the student (just barely) received credit for them based on the diagnostic, retaining this credit is conditional on the student passing tasks that assume knowledge of these topics.

If a student fails a task on a conditionally completed topic, then the system reasons as follows:

This mechanism may be subtle in tasks like multisteps and quizzes, where every question is linked to a different topic, because a student may lose credit for a particular topic (from incorrectly answering a question linked to that topic) while gaining credit for other topics (from correctly answering questions linked to other topics) and passing the task overall.

> Why haven’t I gotten a supplemental diagnostic to place out of more material? I’m doing really well on my tasks.

As described in The Math Academy Way, supplemental diagnostics have nothing to do with student performance. Their purpose is to provide the system with missing information if the knowledge graph shifts around a bit. Students are expected to see supplemental diagnostics rarely if at all.

The original diagnostic exam makes a lot of inferences depending on the structure of the knowledge graph, and if that structure changes even slightly, the system may think “I used to have evidence that the student knows (or doesn’t know) topic ABC, but that evidence was conditional on the previous structure, and now I don’t actually have evidence anymore, so I need to collect some evidence.”

Curriculum

> With short lessons, is the curriculum really comprehensive?

Yes, our curriculum is fully comprehensive – in fact, we perform curriculum comparisons against all the major textbooks to ensure that we’re covering a superset of the material.

How do we cover all the necessary material if the lessons are so limited in scope? By breaking up each course into many, many lessons. A typical course ranges from 150-300 lessons, each lesson containing about 3-4 “knowledge points” of increasing difficulty, each knowledge point consisting of a worked example followed by 2-5 questions of active problem-solving where the number of questions adapts to the student’s performance.

Basically, the “learning staircase” is being chopped up into a massive number of tiny stairs. It still reaches all the way to the top, but the individual stairs are small enough that students don’t get stuck unable to climb a stair that’s too big for them. Each topic is narrow in scope, and they incrementally build up to the deeper abstractions and generalities.

> I passed a lesson, but I don’t feel like I have the deepest level of understanding. Is this normal?

First, keep in mind that each topic is narrow in scope, and they incrementally build up to deeper abstractions and generalities. Students with a high generalization ability may partially extrapolate some of those deeper patterns beforehand, leading to a temporary feeling of incompleteness (e.g., “I’m starting to feel something deeper at play here, but I can’t quite put my finger on it”). Most likely, we do cover that deeper pattern, but it comes later in the curriculum. (However, if you think there’s something we’ve missed, then please let us know specifically what it is! We’re continually refining the curriculum.)

Second, also keep in mind that any lesson you do, it might not feel perfectly intuitive right away, and you might not notice all the connections there are to see. But if you stick with the process and continue periodically reviewing and layering more knowledge on top of that topic, you’ll continually increase your intuition for it, all the way up to a deep level of understanding. The more knowledge you build on top of that topic, the more connections you make, the more deeply ingrained it becomes, the greater your level of automaticity, the more intuitive it feels, the more easily you’re able to see connections to other topics.

Building knowledge is like working out. Like physical transformations, intellectual transformations are produced by accumulating a massive volume of incremental improvements. Our system will imbue you with a deep level of understanding if you’re willing to start at a level where you’re able to solve problems correctly, comfortably, and consistently, and then stick with the process consistently for a long enough time horizon that you could reasonably expect a body transformation if you were physically working out at the gym.

> Will taking a Math Academy course for a standardized exam (e.g., AP Calculus BC) fully prepare a student for the actual exam?

Math Academy courses cover the content knowledge that a student needs to be successful on a standardized test. While this constitutes the vast majority of the work necessary to prepare for a standardized test, it is not fully sufficient. After finishing their Math Academy course, a student must also take a number of practice exams for the specific exam that they are planning to take.

The reason why it’s so important to take practice exams is that – in addition to being timed – standardized exams will “package” the content knowledge within various question framings, phrasings, and general contexts that may initially be unfamiliar to the student. While nobody can predict the exact problems that will appear on the exam, students can train themselves on the same statistical distribution that the exam problems are going to come from. This is accomplished by working through as many practice exams as possible – ideally real exams from the past, or at least practice exams that come directly from the organization who creates the exam. (If such resources are unavailable, it is critical to acquire practice exams from another organization that has a good reputation for matching up its problem types up accurately against the real exam.)

Whenever a student misses a question on a practice exam (or answers the question with a low degree of confidence), they should refer to the solution, identify their mistake, and immediately work it out again correctly. The next day, they should try working out the same problem unassisted. If they solve it correctly, they should wait another several days before attempting the problem again; otherwise, if their attempt is unsuccessful, they should go back to the beginning of this process (refer to the solution, identify their mistake, immediately work it out again correctly, and re-attempt the next day). This is essentially performing spaced repetition on the student’s areas of weakness. This process should continue for multiple rounds through all the practice exams, continuing all the way up until the day before the actual exam.

At the same time, the student should also enable “Test Prep Mode” in their Math Academy course so that they continue receiving reviews on course content instead of being promoted to the next course. It is necessary to continue completing these reviews on Math Academy so that the student does not get rusty on any of the course content. At the time the student takes the exam, they need to be 100% solid and 100% fresh on all their Math Academy course content as well as all the problem types covered in the practice exams.

As a rule of thumb, we recommend that at a minimum, students finish their Math Academy course at least 6 weeks prior to their exam, and go through at least 6 practice tests leading up to the exam. During this time, we recommend about 1.5 hours of prep per day: 30-45 XP of review on Math Academy, and 45-60 minutes taking and re-attempting problems from practice tests (the time it takes to grade the practice test should not be counted towards this time). In the week before the exam, we would recommend increasing the test prep sessions to 2-2.5 hours per day, spending the extra time on practice tests.

The practice tests need to be timed, but they can be broken up into smaller increments. For instance, if an entire test is 2 hours long, then a 30-minute segment can be constructed by doing every problem number that is a multiple of 4. It is important to slice the exam longitudinally like this, as opposed to just taking the first fourth of the exam, because exam problems are often arranged from easy to hard, with hard questions expected to take more time.

Lastly, note that when a student is solid on their course content knowledge and starts taking actual practice exams, they will probably be surprised at how low their score is initially. This is normal and it just takes a bit of exposure to get used to the time limit, question types, and phrasing of the exam questions. Math Academy has extensive hands-on experience preparing students for the AP Calculus BC exam, which is graded on a 1-5 scale (5 being the best), and in our experience, even students who end up getting a 5 often start out getting a 2 or maybe a 3 on their first practice exam. By their second exam, they may get a 3 or 4, and then a solid 4 or maybe just barely a 5 on their third exam, and then a more solid 5 on their fourth exam, and then deeper and deeper into 5 territory on their fifth and sixth practice exams.

> Does Math Academy explain the “why” behind procedures? Are all the concepts taught first?

Math Academy explains the “why” behind procedures all throughout the curriculum. Just to name a particular instance:

• In algebra, when we teach how to solve equations, the very first thing we do is introduce the idea of a solution of an equation: it’s just a number that can be substituted for the variable to make the equation come out true. If you have the equation 2x=6, that’s just saying “2 times something makes 6,” and you don’t even need algebra to know that the solution is x=3 (since 2 times 3 makes 6).

• We first give students some practice solving simple equations like that, without algebra, and only afterwards do we start talking about algebraic manipulations like “start with 2x=6, divide both sides of the equation by 2, get x=3.” That way, students see algebraic manipulations as an extension of the intuitive reasoning that they were using to begin with.

However, concepts and procedures are intermingled throughout our curriculum. We do not teach “all the concepts first” and then “all the procedures after” because it’s not possible to do one in proper depth without the other. Concepts and procedures have a bidirectional, mutually reinforcing relationship. In other words, they build on each other:

• low-level concepts support low-level procedures,
• low-level procedures support higher-level concepts, and
• higher-level concepts support higher-level procedures.

To provide a concrete example:

• a student must be able to count to understand the concept of a number,
• a student must understand the concept of a number to carry out arithmetic procedures, and
• a student must be able to carry out arithmetic procedures to understand the concept of a variable or an algebraic equation.

> Do your university courses have exercises with proofs or is it just computation?

Our Methods of Proof and Discrete Mathematics courses are proof-based. As of the time of writing (August 2025), our other university courses are computation-based, but that’s just because they represent the first course in each subject, whereas a proof-based course would come second. We will eventually be building out those additional proof-based courses, but our current computation-based courses will be prerequisites.

There is sometimes confusion about people thinking Math Academy is stopping at its current level of depth/difficulty, when in reality, we are still building out the curriculum. It is nowhere near finished.

• Does our Methods of Proof course cover epsilon-delta proofs? Yes. Does it cover Papa Rudin? No, because that’s well out of scope. Does that mean we’re stopping short of Papa Rudin? No, it just means we haven’t built up to that yet.

• Another example: we have a computation-based Linear Algebra course that will be a prerequisite for a proof-based Abstract Linear Algebra course later down the road. That second Linear Algebra course will go deeper into the theory and proofs that one might encounter while taking a linear algebra course at an elite university that uses, say, Axler’s book.

  Unfortunately, people sometimes move the goalposts and say “your [first] Linear Algebra course is not as intense as Axler,” when this isn’t even an apples-to-apples comparison. That’s like pointing at a high school calculus class and complaining that it’s not as intense as a real analysis course – of course it’s not! It’s a completely different course; in fact, a prerequisite course; and it’s not meant to cover the same material.

  Axler is really a second course in linear algebra, even if some universities throw students into it as their first course (which ends up causing a lot of unnecessary struggle). We often joke that Axler’s book Linear Algebra Done Right should really be called Linear Algebra Done a Second Time.

• More generally: just because a topic is introduced in a course, doesn’t mean the proof/derivation is also within scope of that course. For instance, formal proofs of derivative rules are out of scope for Calculus I/II (they would go in Real Analysis), and proving the rational roots theorem is out of scope for Precalculus (it would go in Abstract Algebra).

All of these proof-based courses are on our roadmap, but we are building our curriculum from the ground up, we are scaffolding everything to the max, and it’s mastery-based (students are only asked to learn things after having mastered the prerequisites) – so, naturally, we are going to be doing computation-based versions of courses before proof-based versions. The proof-based courses are ultimately just different courses, and we are getting the prerequisite courses in place to build up to them.

> Why does the curriculum focus on building procedural fluency before proofs/derivations? Shouldn’t it be the other way around, prove/derive the formula and then apply it?

In most math curricula, including ours, it’s expected that students will build procedural fluency first and cover full proofs/derivations second, usually in a later course. Why? Because this is the most effective way to get learners to understand the material. You can’t really understand a proof/derivation before you’ve developed procedural fluency. If you try, it will feel like pushing symbols around without really understanding what they mean. Procedural fluency with concrete examples provides scaffolding for learning the proofs/derivations.

In particular, working through concrete examples imbues you with intuition that you will not get if you jump directly to studying the most abstract ideas.

If you go directly to the most abstract ideas, then you might as well be a kid who reads a book of famous quotes about life and thinks they understand everything about life by way of those quotes. You might think you understand the quotes when you’re young, but after you accumulate more life experience, you realize that you really had only the most naive, surface-level understanding of the quotes back then, and you really had no idea what you were talking about.

The way you come to understand life is not by just reading quotes. You have to actually accumulate lots of life experiences. It’s the same way in math. In general, the purpose and power of an abstract idea is that it compresses a zoo of concrete examples. But if you haven’t built up that zoo of concrete examples then you miss out on that power.

> Does the curriculum cover deep understanding? I have been doing a lot of computational problems and I am unsure whether this will lead to deep understanding.

The courses that we cover, we cover comprehensively. But people who don’t know the standard math courses sometimes get confused about what typically goes in what course.

For instance, we sometimes hear people ask why we don’t (e.g.) cover proofs of derivative rules in Calc I/II. They don’t realize that’s beyond the scope of any standard Calc I/II course and is instead covered in Real Analysis, a course that we don’t have yet but is on our radar. The same thing happens with our first course in Linear Algebra. Sometimes people point out that it doesn’t cover all of Axler’s Linear Algebra Done Right, not realizing that Axler’s book is actually meant to be a second course in linear algebra (and Axler even says this himself in the preface).

The courses on our system are comprehensive and we perform comparisons against the major textbooks to make sure we’re covering all the bases. There is definitely room to include projects and challenge problems, which are on our radar – but we have yet to hear a specific example of a piece of knowledge that is properly scoped to one of our existing courses and does not appear in said course. If anyone presents such an example, then we will of course add it.

Personally, every time I have engaged in discussion about this, it has always turned out that

1. the critic does not understand the standard scope of math courses, or

2. their critique is not of Math Academy but of the standard scopes themselves, or

3. the critic has some nebulous “feeling” that we should cover more but can’t articulate a specific concrete example, or

4. the critic doesn’t want to put in the work to shore up their skills (especially if they did poorly in the diagnostic) and prefers some other resource that waters down material to make it accessible with fewer prerequisites.

(A concrete example of #4: I spoke with someone who, upon starting our calculus content, claimed we didn’t cover the concept of the derivative as a rate of change in physical scenarios like the height of water while filling a trapezoidal pool, when in fact we have a bunch of topics on those physical/geometric derivative problems that come later in the course once you’re able to interpret the derivative as the slope of the tangent line, compute some basic derivatives, and interpret the meaning of f’(x) given a verbal description of f(x).

He wanted to just go to the most advanced content right away without filling in his prerequisites, and when we placed him at the level that he was truly at, he felt like it was too elementary and mechanical-skill-based to pave the way to deep understanding. Basically, he just didn’t want to do the work, and instead wanted to read about the derivative in broad strokes in a complex setting, without being made to solve actual problems.)

> It’s hard to believe that 5 hours a week for a year starting from basic multiplication tables will have me completely prepared for university courses. I’d prefer an explanation for people who are not familiar with the XP system.

We realize that this can be a bit shocking! For this to feel more realistic, it’s important to first understand that not all the math that children cover in school is necessary for university math. As described in The Math Academy Way, we developed a Mathematical Foundations course sequence specifically for the purpose of getting adults up to speed as quickly and efficiently as possible with all the prerequisites that they would need to know (fractions through calculus) for university-level math courses. Roughly a third of topics in our Traditional sequence are required by school standards but do not actually come up as prerequisite material in university-level math. Those topics have been stripped out of the Foundations sequence.

Now, knowing that the Foundations sequence covers about two-thirds the content in the Traditional sequence, the rest of the argument follows from the success of our original in-school program in Pasadena where 6th graders started at various places in Prealgebra, did about 40-50 minutes of fully-focused work per school day for the next 3 years, and covered all of Prealgebra, Algebra 1, Geometry, Algebra 2, Precalculus, and AP Calculus BC, passing the AP exam by the end of 8th grade. While this may also seem shocking, Math Academy has the AP scores to prove it, and there has been plenty of news coverage over the past decade.

Now, look at the numbers: 40-50 fully focused minutes per school day × 180 school days year × 3 years comes out to about 24000 minutes or 400 hours, and our Foundations series is about two-thirds the size of that (since roughly a third of topics are not actually prerequisites for university math), which comes out to about 267 hours. Divide by 52 weeks in a year, and you’re at about 5 hours per week.

> Why is “holistic mode” (in which students also fill in any missing knowledge in lower-grade topics that are not prerequisites of their enrolled course) disabled for university courses?

Students are most likely to succeed when they break up long-term goals into short-term goals, maintain momentum, and experience plenty of small wins along the way. Suppose an expert tutor works with a Linear Algebra student who makes the following request:

In this situation, the expert tutor should try to dissuade the student:

If a student gets as far as university level math but has missing background knowledge, it would be a mistake to front-load that missing knowledge and fill it all in while they complete their first university course. It would reduce friction and increase motivation to instead spread out the work, which is what will happen naturally as the student takes more university courses in non-holistic mode.

> When you introduce additional scaffolding to increase pass rates of lessons, how do you know the increase in pass rate actually represents learning? Couldn’t the pass rates increase simply due to greater priming?

It’s something to watch out for – but we do watch out for it. We also track review and quiz performance. There is less priming for review tasks, and no priming for quizzes. If students were to bomb those questions on quizzes, that would signal that the learning was superficial or temporary.

Additionally, most topics in our system have many post-requisites, so students are continually made to layer more advanced skills upon what they’ve learned. If they didn’t actually learn a prerequisite, they wouldn’t be able to continue executing progressively more advanced skills on top of it – just like if a basketball player can’t dribble the ball, they won’t be able to successfully complete any plays that involve running across the court with the ball.

MISCELLANEOUS

Math Academy Itself

> Why use Math Academy instead of self-studying a textbook or free online resource?

Math Academy’s key value proposition is that it maximizes every student’s learning efficiency. While it is possible for sufficiently motivated students to learn math by self-studying a textbook or free online resource, there are many sources of inefficiency:

• Not hyper-scaffolded. Students will periodically run into situations where they are confused about a logical leap that has taken place. It often takes a long time to resolve the confusion and figure out the logical rationale (if the student figures it out at all).

• Doesn’t track student knowledge and implement mastery learning (i.e., does not ensure that the student has mastered the prerequisites before moving on to new material). Students will feel a large gap between their level of knowledge and the new material, which leads to more confusion and time wasted trying to figure out what prerequisite knowledge they are missing and how to learn it. Often, students will be unable to pinpoint all their missing prerequisite knowledge and will consequently be unable to fully grasp new material, even if they grasp it partially.

• No spaced review. Students will quickly become rusty on the material that they learn. Not only will students come out of their course of study having forgotten much of the content, but even during the course, they will constantly be forgetting the prerequisites for new material that they attempt to learn.

• Doesn’t adapt to the student’s level of performance. Students waste a lot of time doing the wrong amount of work. Sometimes a student will grasp a topic quickly and do far more practice than is necessary; other times they will struggle with a topic and not get enough practice to reach mastery.

• Leaves the definition of mastery open to interpretation by the learner. It is difficult for a student to know when they have mastered a topic well enough to continue moving forward. Even in good faith, students often think that they have learned a topic well enough when they actually haven’t – and they will not realize this unless their mastery is being evaluated by an expert. On the flipside, students can also take things too far in the way of perfectionism, spinning their wheels on the same topic for days or when there is a minor point that doesn’t make perfect intuitive sense, when it would be more productive to keep moving forward and solidify their understanding by building on top of it.

This list could be continued endlessly with other items discussed earlier in the body of this book, but the point is that all of these sources of inefficiency introduce unproductive friction into the learning process, lowering a student’s educational progress per unit time and effort that they put towards learning.

Math Academy removes as much of this learning friction as possible, maximizing student learning efficiency. That is our main proposition: sure, it’s possible to learn math elsewhere, but it’s way more efficient with Math Academy.

It’s worth noting that efficiency is important not only because students make faster progress, but also because they are less likely to quit. Typically, people get off the train and stop learning math once it begins to feel too inefficient relative to other opportunities in life. In anything one does, once the progress-to-work ratio becomes too low, one will lose interest and focus on other endeavors where their progress-to-work ratio is higher. Efficiency keeps that progress-to-work ratio as high as possible, keeping students on the math learning train as long as possible.

> Why isn’t Math Academy free?

Math Academy requires payment because it takes so much time & effort to build. Additionally, it must be priced in a way that the company’s solvency is not dependent on a massive user base.

Math Academy is intent on using the most effective training techniques, but most people are not that serious about their learning. Maximum-efficiency learning feels like a sweaty, exhausting workout with a personal trainer, for at least several hours spread across several sessions each week.

When an education company depends on a massive base of learners, most of whom are not serious enough to engage in that level of intensity and frequency in their training, it requires the company to employ ineffective learning strategies that do not repel unserious students. The company must convince their students that they’ve managed to learn things despite putting in little to no work. (This can be accomplished, for instance, by cherry-picking the simplest cases of each topic and letting students move on despite poor performance on prerequisite material.) Unlike such companies, Math Academy is in the business of optimizing real learning – not just the perception of it – for students who are willing to put in serious work.

At the same time, of course, we do want to make mathematical talent development accessible to more and more people. As discussed in The Math Academy Way: before Math Academy, if a student wanted to replace their traditional schooling with the equivalent duration of 1-on-1 coaching from a personal trainer who develops their mathematical talent using a personalized training program that is tailored and constantly adapting to their individual needs, they would have to obtain it from a private tutor for a typical price of at least $50/hour. Year-round talent development, with a daily work time that is in line with the amount of time that students would be working anyway during the school year (conservatively, 1 hour per weekday) would cost $50 × 5 days/week × 52 weeks/year = $13,000/year.

Bringing that figure down to $499/year (26x cheaper) via Math Academy makes mathematical talent development accessible to many, many more people. That’s not everyone, and there are still people who are priced out, but providing a 26x cheaper option is a good starting point towards a goal of making mathematical talent development accessible to more and more people.

> Math Academy maximizes learning efficiency if a student is willing to engage in forms of training that are highly effortful. What about for students who don’t have as much energy and motivation?

Math Academy teaches math as though we were training a professional athlete or musician, or anyone looking to acquire a skill to the highest degree possible. When a student signs up for Math Academy, it’s like going to a gym where one of the personal trainers was a former Olympic sprinter, and telling them “I’m going to show up 40 minutes per day, 5 days per week, and I want you to use whatever methods of training are going to make the most improvements on my 100-meter dash time. I don’t care how exhausting they are; I am willing to work hard.”

Many students do not want to devote this much effort to their learning, just like many people who sign up for the gym do not want to do an Olympic-intensity workout most days of the week. While it’s true that willingness to work hard is a bottleneck for many students, such students are not part of our target market. If a student is not willing to put forth a high degree of effort engaging in the most effective training techniques, which are taxing, then the Math Academy system is not a good fit for them.

> Where do the exercises and content on Math Academy come from? Are they all made in-house or pulled from other materials?

All of our content and exercises are created in-house, carefully crafted over many years by a team of math experts. We perform curriculum comparisons to ensure that our content is comprehensive, but everything on Math Academy is created in-house.

Features that Do Not Exist for a Good Reason

> I don’t really want to do any of the learning tasks that Math Academy presents to me. There are other topics I would rather learn. Why can’t I choose my own tasks?

Math Academy’s main value proposition is maximizing student learning efficiency. That is our top priority. When a student signs up for Math Academy, we are making a promise to them that their learning experience is going to be as efficient as possible. The student is going to learn the most math possible in the time that they’re devoting to study.

In order to keep good on that promise, we have to use a lot of sophisticated algorithms to analyze the student’s knowledge profile and select their tasks. The whole system has been built around that concept.

We do have some ideas for features that will give students more agency over what they’re learning, but it’s going to take some work because we have to be careful not to allow students to make decisions that throttle their learning efficiency. The approach that we’ve been thinking about is less like “select whatever topic you want at any time” and more like “tell us what your specific goal is and we’ll put you on the most efficient path to that goal.” Of course, none of that is fully-baked yet, but it’s something that’s on our mind and that we’re working on.

> Why can’t I edit my knowledge profile?

Learners have a tendency to massively overestimate self-reported knowledge, and then fault the resulting instruction for moving too quickly, not explaining enough, or otherwise being too challenging, when the issue is really that they lack sufficient mastery of prerequisites. To construct an accurate knowledge profile, the system must infer it from a student’s demonstrated ability to solve problems.

> Why don’t you provide information about what topics are going to be on a quiz? Surely students would do better if they knew what was going to be on it.

The point is to get an honest signal of whether a student is able to solve problems without priming beforehand. Students will of course do better with priming, but that would no longer be a true assessment of their ability to solve problems unassisted.

If a student can only solve a problem after being reminded how to do it, then they don’t actually know how to do it independently. (It’s like they’re lifting weights at the gym but they can only lift the weight with the help of a spotter.)

And if they can solve a problem without a reminder, then being given a reminder robs them of the practice that would otherwise improve their retention of the material. (It’s like a spotter unnecessarily stepping in to assist a lifter, and reducing the weight even further below what the lifter could successfully lift on their own.)

> The problems feel tedious and I keep making silly mistakes. Maybe you could have a setting that differentiates between students that need practice to take tests and those who just want to learn concepts?

We teach math as if we were training an aspiring professional athlete or musician, or anyone looking to acquire a skill to the highest degree possible. This isn’t edutainment, this isn’t enrichment, this isn’t a “math appreciation” course. We expect our students to actually master the material and develop as strong a command over math as a musician’s command over their instrument. If that’s not what you want to get out of your math learning, then Math Academy probably isn’t a good fit for you.

But if mastery is what you want to get out of your math learning, then it’s important to realize that climbing a skill hierarchy like math is not just about conceptual understanding. It’s also about reliable execution – and a high frequency of silly mistakes indicates that you need more practice with the material.

Why? Because if you don’t clean up your silly mistakes on low-level skills, then you eventually hit a wall where no matter how hard you try, you’re unable to reliably perform advanced skills due to the compounding probability of silly mistakes in the component skills. Think about gymnastics: if you’re “almost” able to land a backflip, then that’s great… but at the same time, you’re not ready to try any combo moves of which a backflip is a component. Even if it’s a silly mistake keeping you from landing the backflip, you still have to rectify it. (And this is the most optimistic scenario – other times, silly mistakes indicate a deeper conceptual misunderstanding that you don’t even know you have until you are held accountable for rectifying those mistakes.)

> Why isn’t there an “I don’t know” button on questions during tasks other than the diagnostic?

If a student ever gets stuck during a lesson, then they always can and should go back to the preceding worked example and follow along carefully to identify what they missed when they read it the first time. If a student ever gets stuck during a review or multistep, then they always can and should go back to the corresponding lesson in reference mode. The information that a student needs to solve the problem should always be there.

For instance, review questions are pulled from the same exact pool of questions that a student might see during the lesson. So, any review question that a student receives will line up with one of the question types covered in a worked example in the lesson. (To be clear: the review question will be chosen as a different question within that pool – the student will not be given a review question that is exactly the same as one they already answered during the lesson.)

A student should never be in a position where they are being asked to perform a skill that’s not covered in the supporting lesson, so we do not wish to communicate otherwise (which is what an “I don’t know” button would do).

That said, it is true that students should not refer to reference material during quizzes, so why isn’t there an “I don’t know” button on quizzes? This is something we considered when first implementing quizzes – but, working with a large number of students, we’ve experienced that many students will abuse an “I don’t know” button if it’s provided.

• This can be intentional, e.g., adversarial students (especially kids who are using the system for school and have a mentality that is not fully aligned with the learning process) will click “I don’t know” simply to avoid doing work. When we first deployed the automated system in school classes, there was a period of time where the system was getting attacked left and right by adversarial students trying to game the system (or otherwise create chaos that they could leverage to confuse their parents and get out of doing work). It took a lot of effort to patch up exploits, and whenever we make adjustments to the system, we’re always on the lookout for any ways that it can be exploited (because if it can, then it will, and the behavior will spread).

• Or it can be unintentional, e.g., underconfident learners may underestimate their ability and give up too early. When a tutor is working with a student on a problem, it is not uncommon that a student will claim not to know how to do the problem, but when the tutor asks the student to make their best guess, the “guess” is correct – and when the tutor asks the student about their thought process afterwards, it turns out that the student knew how to solve the problem, but they weren’t confident about it and they didn’t want to risk getting it wrong.

While it may be subtle, removing the “I don’t know” button is a crucial safeguard to protect many students against self-destructive behavior. In theory, no such safeguards should be necessary, but in practice, a vital component of a functioning learning system is that it must be robust to all sorts of unexpected behavior arising from the various human emotional experiences associated with learning and intense training. Often, these emotional experiences can be intense and (if the option is provided) lead people to make short-sighted decisions that ultimately hinder their educational progress.

> Why doesn’t Math Academy use large language models (LLMs) to engage students in conversational dialogue?

Many people who have (unsuccessfully) attempted to apply AI to education have focused too much on the “explanation” part and not enough on scaffolding, navigating, and managing the entire learning process.

It’s easy to go on a wild goose chase building an explanation AI. You fall in love with the idea of AI having conversational dialogue with students, and then you get lost in the weeds of complexity. You solve just enough of the problem to produce a cool demo, yet you’re still hopelessly far away from self-service learning in real life.

Dialogue isn’t even necessary. We simply hardcode explanations into bite-size pieces, served at just the right moment. And we close the feedback loop by having students solve problems, which they need to do anyway. (Their “response” is whether they got it correct.)

Sure, hard-coding explanations feels tedious, takes a lot of work, and doesn’t have produce the same “wow” factor as an AI that generates responses from scratch – but it’s a practical solution that lets us move on to other components of the AI that are just as important (i.e., the entirety of this book). Just to name a few such components:

• After a minimum effective dose of explanation, the AI needs to switch over to active problem-solving. Students should begin with simple cases and then climb up the ladder of difficulty, covering all cases that they could reasonably be expected to solve on a future assessment.

• Assessments should be frequent and broad in coverage, and students should be assigned personalized remedial reviews based on what they answered incorrectly.‌

• Students should progress through the curriculum in a personalized mastery-based manner, only being presented with new topics when they have (as individuals, not just as a group) demonstrated mastery of the prerequisite material.

• Students should progress through the curriculum in a personalized mastery-based manner, only being presented with new topics when they have (as individuals, not just as a group) demonstrated mastery of the prerequisite material.

• After a student has learned a topic, they should periodically review it using spaced repetition, a systematic way of reviewing previously-learned material to retain it indefinitely into the future.

• If a student ever struggles, the system should not lower the bar for success on the learning task (e.g., by giving away hints). Rather, it should strengthen a student’s area of weakness so that they can clear the bar fully and independently on their next attempt.

> If I get stuck, is there somewhere that I can ask for help or a further explanation?

We don’t offer human tutoring services. However, we’ve been quantitatively analyzing and refining our content for years, smoothing out sections where anyone has struggled. Thousands of learners have successfully made it through our courses, and on average, students pass lessons 95% of the time on the first try and 99% of the time on the second try.

Furthermore, on the rare occasion that a learner does happen to get stuck and fail a lesson twice in the same place, the system will automatically have them review the prerequisite knowledge that is most relevant to their area of struggle before having the student re-attempt the lesson.

> After I started a quiz, I had to pause to do something else. I came back a while later and it said my quiz was completed and the answers I didn’t get to were marked incorrect. Is there a way to retake it?

No, there isn’t. You should only begin a quiz when you know you have the time to take it. You have about a 30XP window on your dashboard for when you have to take the quiz, so there’s some room to do other things if you need to in the meantime.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.