CS Primer Show #23: MathAcademy and the efficient pursuit of mastery
Math Academy was originally built to support a school program. How come it also works so well for adults? What makes someone a student a good fit for Math Academy -- what's required to succeed? The idea of calibrating to student interest/motivation profiles in the future, just like we currently calibrate to student knowledge profiles.
Cross-posted from here.
Want to get notified about new posts? Join the mailing list and follow on X/Twitter.
Below is a smoothed version of the raw transcript.
Oz: Justin, I was excited to talk to you because you guys have done something that seems impossible. If you told me there would be a website or a system where adult learners would be excited to consistently solve math problems—problems they probably should have learned in high school or maybe early undergrad—voluntarily at the age of 30 or so, I would have been skeptical. And yet, people are doing this every day, excitedly telling their friends about it.
By the way, several people have suggested we have you or someone from the Math Academy team on the podcast multiple times. Thank you, everybody, for the suggestions. If you’d told me all of this a year ago, I would have had a hard time believing it. But it seems like you’ve done it. There’s something about Math Academy that draws people in. How has this come to be? What do you think is the quintessential thing about Math Academy that has made it so successful?
Justin: Yeah, well, let me start by saying I’m also very surprised by all the adult learners solving math problems consistently every day. They’re so into it, posting their progress on Twitter, encouraging each other. This has been kind of a surprise because we started out as a school program, building software to support it, and it was just kids on the system. We never really thought it would take off with adults, too.
But I guess lots of people want to learn math for lots of reasons, and there are plenty of adults in that boat. In terms of what makes it work well for adults, a couple of things come to mind. The biggest factor is that adults are very time-constrained. Efficiency, particularly learning efficiency, is key.
Everybody likes learning efficiency—it’s good for everyone—but kids generally have a lot more time to waste than adults. When you’re an adult trying to learn an academic subject, not only are you time-poor, but you’ve probably spent your whole day doing cognitively taxing things. You’re not going to commit to something unless it’s efficient enough and gives you enough bang for your buck to make progress.
This has been a struggle for many adult learners in the past, especially with math. You see this in Reddit threads all the time, where people say things like, “I want to learn machine learning, but I don’t know the math behind it.”
Charlie: How do I learn the math? It’s constantly my problem. I feel like I have to go back to arithmetic in order to properly do it.
Justin: Right. You ask on one of those threads, and they tell you, well, you’ve got to know this, that, and the other thing. You end up with a stack of 15 textbooks, and probably only seven of them are actually necessary. Some people even say you need group theory and abstract algebra for machine learning—don’t listen to that. But you’ll still end up with this stack of textbooks.
Then you start working through them, and after a month, you’re 50 pages into the first one, which is 400 pages long. Then you do the math and realize it’s going to take you eight months to finish just the first textbook. Multiply that by seven, and suddenly, you’re thinking about the next decade of your life. At that point, you kind of give up.
Oz: To fill in anyone who hasn’t done Math Academy—and that’s not me anymore because I signed up for Math Academy last week to do some research—
Charlie: I just signed up an hour ago. I got to the diagnostic and thought, I’m going to need more time.
Oz: Time diagnostic, Charlie. Get back to it. To fill people in, the way you can avoid the stack of 15 textbooks—even though theoretically many of those topics are on the way—is through a diagnostic test.
If it’s granular enough, it narrows the set of topics. For example, it might not cover all of calculus but instead focus on specific aspects you need to know and don’t already, at least according to the diagnostic test. Once you’ve narrowed that down, you can figure out how much work to do per day and estimate how long it’ll take to reach a milestone.
This gives you clarity in a way that a Reddit thread saying, “Start from middle school and do everything,” doesn’t. You directly know what you need to learn, and it takes as long as it takes. The diagnostic is an attempt to narrow it down and provide clarity on where to go from there.
Charlie: The diagnostic plops you into a knowledge graph that looks like it goes from fourth grade through graduate school. Is that the idea? It can appropriately place me based on which individual concepts I need extra help with. Is that what the diagnostic is doing?
Justin: The diagnostic basically places your knowledge profile on the graph. You take a diagnostic for a particular course. It’s not like we assess you over the entirety of math. We kind of expect that you generally know where you are—you’re not confused about whether you’re ready to study multivariable calculus or need to learn how fractions work.
At least for now, we expect you to enroll in the appropriate course. But we do check for prerequisite knowledge you might be missing. For example, if you sign up for a calculus course, we’ll look back to see if you know trigonometry and algebra. If there are gaps, we’ll detect them, and that’s fine—we’ll add those to your learning plan.
It’s about tracing your knowledge profile in a section of this big graph of mathematics. Currently, it starts at fourth grade, and the highest levels we have right now are methods of proof, calculus-based probability and statistics, multivariable calculus, and a math-for-machine-learning course that includes a lot of university-level math.
We’re continually expanding the curriculum. It’s going to cover all of an undergrad math major and add actual machine learning courses. We’re currently developing a machine learning course that teaches algorithms, including backpropagation, with students working problems by hand. We’ll also have a second course that gets into transformers and similar topics. We’re continually expanding this graph.
Oz: Everything in education feels like a trade-off, right? If you go down the track of making it as efficient as possible, you need to trade something else off. Otherwise, every program would just be as efficient as possible.
What do you think is the trade-off that Math Academy makes, or what is the audience for whom it’s just not going to be a good fit?
Justin: The trade-off is that the people for whom it’s not a good fit are those who want a low-effort, mentally easy, just-for-fun experience. Imagine someone who goes on YouTube and says, “I want to learn some interesting math,” but what they really mean is they want to watch someone talk about interesting math with pretty shapes on the screen. They don’t want to work out problems or put in effort; they just want their brain tickled a little. That’s not who we are.
It’s also not for people who want to solve one or two math problems a day during their train ride to work or something. The way we phrase it is that we focus on mathematical talent development. We train our students using techniques similar to how a coach would train an aspiring professional athlete or musician. It’s serious training.
It’s not for someone who just wants to appreciate math or dabble in it. It’s not a math appreciation software. It’s like an actual sweaty workout. You need to come to it with some reason or motivation.
Charlie: Is that what you’re saying?
Justin: I don’t know that it has to be intrinsic because plenty of people want to learn math in service of other goals. For instance, with machine learning, people might have intrinsic motivation for coding or find AI in computer science cool, but then realize it involves a lot of math they didn’t like in school. They might think, “Oh, fine, I guess I have to do this math.”
Sometimes that softens, and they think, “This actually isn’t so bad. Maybe I like it a little.” Or they find some beauty in it. But it doesn’t require intrinsic motivation. It just requires consistency and commitment, like working out at a gym.
You don’t have to be intrinsically motivated to work out. You might set up some extrinsic structure, like wanting a beach body for summer. There are different reasons that can get you into the gym. As long as you’re there, doing an effective workout consistently, positive changes will happen.
Charlie: I’m envisioning this like a skill tree in a video game, like Diablo II, where I start as a weak warrior but can become a god-like Paladin at the end. If you follow along, you can fill this thing out. Do you show the trajectory from a UX perspective? Like, “You are here on your larger journey against this map,” or is that too much info?
Justin: Currently, we show a knowledge graph of the course. You can see, maybe there are 300 topics in the course, and as you complete topics, they fill in. But that’s still just a small subset of our entire graph.
One thing we want to improve in the future is putting more leverage behind our knowledge graph visualization. Not just showing the course, but showing where you are in all of mathematics. We want to show a time lapse: “Here’s where you were a month ago, and here’s all the tasks you’ve completed.” Your knowledge graph would fill in as you progress.
We’d also show how every task you do hits specific topics in the graph and which topics receive implicit review from practicing those sub-skills. If you take too much time away from the system, we could show your knowledge decaying, peeling back, or fading.
We want to visualize this as a representation of a student’s mathematical brain—almost like an MRI of their math brain. We definitely want to lean more into this, including gamification, where you can level yourself up.
Charlie: It feels like a fingerprint. Everyone has their unique fingerprint, and this is the fingerprint that led me to this particular level of mastery.
Justin: Exactly. It is like a fingerprint. That’s one of the reasons why typical classes struggle. In a typical school, you have 30 kids in the same class, and each has their own fingerprint of a knowledge profile. They’re all missing different pieces of prerequisite knowledge. Somehow, the teacher is supposed to deliver one lesson that magically calibrates to all these individual fingerprints. It just doesn’t work well.
That’s why this approach is so efficient—every decision is calibrated to that fingerprint.
Charlie: I feel bad I haven’t fully done the program yet. I’m going to. What is the core experience? Is it solving problems that are perfect for you in this moment?
I think about other examples, like Khan Academy, where the core thing might be charismatic whiteboarding videos. What is it about Math Academy that’s the hook? Is it problem-based learning with spaced repetition, where you’re always working on something perfectly hard for you?
Justin: That’s exactly it. Every moment a student spends on the system is intended to be working on something at just the right level to help them move as fast as possible. If it’s too easy, there’s no point. If it’s too hard, there’s no point. You want it right at the optimal level of difficulty.
That might mean they know the prerequisites for a topic and are ready for a new lesson, or it could be reviewing information or problems they’ve done before, but their memory has faded a bit. Whatever it is, we’re always trying to use the student’s time as effectively as possible.
Oz: That’s it. Can I add something to that characterization? I think I’m coming both in favor of Math Academy and also against it for those who don’t resonate with this.
From what I’ve seen so far, the basic structure is you have a worked example, maybe two, and then two questions. Then another worked example, and then questions. This is the overall structure.
Worked examples are fantastic. There’s very little solid support for anything in educational research, but there’s a lot of support for worked examples. Seeing someone step through how to solve a problem helps you solve similar problems. Providing a worked example followed by a question is a good overall loop: “This is how you do this kind of thing; now you do this kind of thing.”
This is very different from the Khan Academy approach. Parts of Khan Academy do follow that structure, but other parts are more about exploring interesting ideas in a general space, connecting them to other topics, and so on.
Math Academy, from what I’ve seen exploring it for just a few hours, isn’t about providing motivating context or showing how examples connect together. It’s not about explaining why you might be interested in this topic.
Instead, it immediately jumps to, “Suppose you want to take the derivative of an inverse trigonometric function. Here’s how you do that. Now you go do that.” It assumes you already have the interest in solving this kind of problem and, without any fanfare or inefficiency, directly shows you how to solve it, and then you solve it.
Charlie: Okay, that’s really good because I wanted to ask about this. I used to grind integrals in high school, and I could crush them. I couldn’t do that now, and I was thinking back to it. I was like, well, maybe if I knew—and I don’t even know if this is true—but integrals are about measuring the volume of water in jars or something, maybe I would have retained that in some way. I think that’s probably just fantasy, that having that additional context would have helped me retain it. But it sounds like you’re not providing why you’d want to learn derivatives or integrals or giving that kind of context.
Justin: The context comes, but it typically comes after the baseline mechanical skills have been laid. More context is also coming in the future that isn’t currently there. Let me elaborate.
For the mechanical skills, let’s take that calculus course as an example. We have a lot of problems and topics that involve using integration to measure volumes of things—like the volume of water with an integral, or other concrete meanings of integrals in many real-life scenarios. But it’s true that we don’t start with a real-life scenario when teaching integration.
We start with the most simplified mathematical idea of just the area under a curve and gradually expand from there. The reason is that we’ve been building the system to focus on raw learning efficiency—building on prerequisites. If you try to motivate a subject with a real-life scenario but the student isn’t ready to handle it yet because they lack prior knowledge, you risk overpromising.
For example, if you say, “By the end of today, you’ll do something amazing with this cool math you’re learning,” but they only learn a small piece of it and it’s not as amazing as promised, it can be disappointing. The amazing part does come, but only after more practice.
Additionally, the connection between something like measuring the volume of water and integration isn’t self-evident to students. It often requires a lot of explanation, especially if a student is struggling. This can lead to students zoning out. There are many opportunities for pedagogical mistakes if you’re not careful.
Expert teachers are good at this. They can introduce concepts in an engaging manner without spending too much time explaining connections. They don’t overpromise, but they use interesting examples to maintain interest while moving students along efficiently.
This is something we want to integrate more into our system in the future—like having a conversation with a really cool tutor who makes you excited about the subject. That might involve historical anecdotes or applications that interest you.
Oz: Maybe this is the interesting thing to me, where you guys have really made the most of the medium. If the internet or a software system is very good at calibrating an exercise, providing a canned worked example, and giving dynamic feedback like, “Here’s the next good question for you,” maybe that’s what you should focus on and do very well.
To an extent, you can expect users to bring their own motivation or motivating context. This is the system where, if you don’t need that extra motivation, you do very well. You’ve carved out that niche.
In a way, the medium plays into this. People are very excited about Math Academy, and people are also very excited about Grant Sanderson. He does the inverse of what you do. There’s no practice or problems, but it’s exciting and provides the motivating context. He makes the most of YouTube. The recommendation algorithm keeps feeding people who want exciting math videos his content.
It’s a nice balance. He focuses on making math exciting and accessible, while you focus on the problem-solving part for those who are already motivated. Maybe that’s a good thing. You don’t necessarily need to change that.
Maybe you should just put a sticker on the front of it: “This is like a gym where you come ready to train.” Evidently, there are already a lot of people in that mindset, waiting for someone to directly give them the sequence of problems they need to get into something like machine learning.
Charlie: That works for adult learners, but it also works for school systems where there are teachers and external structures. These externalities mean you have to adhere to them.
Math Academy is serving a lot of different populations, and I can imagine the product forking based on the differences between adult learners and the institutional education space. In the latter, you might have to build more teacher tools.
For example, how do I manage a classroom of students using a flipped classroom or blended learning approach? Teachers need a bird’s-eye view of what students are working on and where they stand.
I feel like you could go in a lot of different directions. I used to work at a company doing similar things 10 years ago, and teacher insights ended up being a big area where we spent a lot of time.
Justin: That is also on our radar. We’ve actually taught some classes on the Math Academy system. We had an elementary dashboard monitoring student progress and everything. We’ve got a couple of schools using the software, and it’s definitely going to be more of a thing in the future.
Ultimately, we want to become the ultimate math learning platform. A key ingredient for that is layering things on, starting with learning efficiency. We’ve optimized efficiency to a point where it feels close to the maximum—it’s hard to squeeze much more out of it.
Now, the focus is on keeping people excited about doing the work and staying consistent. Incentives, gamification mechanics, and contextual elements like historical anecdotes could be very motivating for many learners.
Of course, we wouldn’t want to clutter the experience or slow down the learning. It would require a very delicate balance. We definitely wouldn’t want to go to the point of being like a YouTube video, but there’s probably some dose of it that could work.
Oz: That’s the interesting question, right? What I was hinting at earlier is that there’s always a compromise in education. How do you trade off with efficiency? If you start catering to that other need, can you do it without compromising what’s already working?
One thing I came to understand better as I taught more is that the motivating context for one student is very different from that of another. I thought I was getting better at delivering motivating context, but in reality, I was just getting better at catering to a particular group of students who resonated with it.
Over time, I developed a more historical role-playing approach in class, particularly for esoteric systems concepts. For example, why is the virtual memory system we use in Intel-derived architectures the way it is? Without the historical context, it’s hard to understand the complexity or even why you’d bother learning it.
We’d role-play as if it were the 1980s and imagine solving specific problems, like providing memory protection on a system. Students would provide ideas, and we’d explore trade-offs, leading to the current system. After repeating this process five times, we’d arrive at what we use today.
I thought this approach was fantastic. Of course, I rated my own teaching highly. For some students, it helped them finally understand the concept. But for others, it felt like we spent an hour developing motivation they already had.
Some students were excited to learn because of the role-playing, while others just wanted to learn it because it had come up at work or was the next thing in the textbook. Those students didn’t need extra motivation.
I thought I was getting objectively better at teaching by designing the classroom experience I would have wanted myself, combined with feedback from students who resonated with it. But as I listened to feedback from other students, I realized it was fine—or even better—before.
I’m sharing this because I’m not sure this is something resolvable. Is the best approach to find a reasonable solution for everybody, or should you just put a sticker on the front saying, “If you are this kind of person, this is the perfect product for you”?
Justin: Here’s the thing: I don’t think it has to be one-size-fits-all. The dose of contextual information, historical anecdotes, or whatever else can be adapted. If you’re a human teacher, you either include it in the lesson or you don’t. Different learners have different motivational profiles, just like they have different knowledge profiles. You’re either matching or not matching to those profiles.
In an adaptive learning system that’s completely individualized, maybe for some students, the best motivational approach is to strip out all historical anecdotes because they don’t want that. They just want to know how to do the problems and move through them as fast as possible.
Oz: If you could solve that problem, that would be very interesting in itself. Could you give someone a diagnostic where, at the end of it, you understand their—should we call it learning style? Maybe we should call it something else.
Justin: Motivational profile, maybe. Or motivation style—something like that.
Oz: If you could give them questions like, “Is it interesting to you that trigonometry came from measuring the distance to the moon or the height of mountains?” you could determine their degree of interest in that kind of thing—their interest profile.
Justin: I think that’s something we definitely see on our roadmap in the future as we build out more of the product. There are other things coming first, though. For instance, we’re working on a streak mechanism right now to help learners stay consistent with using the product. But I’m excited to try to get our arms around the problem of motivation profiles eventually.
Charlie: Can I ask about the map a little bit? It feels like every example of adaptive learning that works really well is in the mathematics space. The concepts are atomic enough to have well-defined problems around them. You can clearly map the prerequisite relationships between these atomic concepts.
In my own experience, it gets fuzzier when you move to other subjects. The lines are blurrier, and it’s harder to know exactly how to tailor content at that atomic level while also building a prerequisite tree.
Have you all thought about applying this to other subjects? I know it’s called Math Academy, obviously, but would you ever consider a rename?
Justin: The grand plan is to expand outward from math, but there’s a limit to how far outward we’ll go. I don’t see us doing history courses or English literature anytime soon. Our focus has been math for years.
Recently, we’ve started expanding into math-adjacent areas like computer science, including machine learning, and we’ve begun working on an intro to programming course. We’ll also be doing a lot of applied math, such as quantitative finance and general physics.
The areas where it gets a little fuzzier are the less inherently math-focused sciences, like biology. You can make biology very math-focused, and maybe we’ll have a quantitative biology course at some point for those interested. But would we ever have a course like cell biology? I don’t know.
Oz: I think it’d be great to have a cell biology course. There’s a fantastic book called Cell Biology by the Numbers, which argues that a lot of molecular biology is taught too conceptually. It suggests using back-of-the-envelope calculations to build intuition about what’s actually happening.
The book is based on questions like, “How many proteins are in a cell?” or “What’s the flow outside the cell?” It’s about calculating things using basic numbers. The math isn’t complex, but the quantitative approach to biology is really interesting.
Justin: That’s cool. I agree. I’d probably enjoy taking that kind of course—almost like a mathematical applications to biology course. Or maybe it could show up as applied projects tacked onto various math courses.
Oz: There’s an interesting dynamic there. I wouldn’t call it a conflict, but maybe a complementary approach.
Justin: Are you talking about Execute Program?
Oz: Yes.
Justin: I’m familiar with it, but I haven’t actually done it.
Charlie: I found out I was paying for it for the last six months without using it, which I’ve recently resolved. Now, I can do the same thing for Math Academy.
Oz: Execute Program is a similar approach to programming, and it’s very good. It’s different from something like Primer or other resources in computer science. There’s this spectrum from legible to illegible, or structured to unstructured. The more structured you get, the more you can build systems around things like knowledge graphs and spaced repetition.
Gary is very into knowledge graphs. He puts them front and center. I’m not sure if it’s on the homepage, but it’s very easy to see the knowledge graph guiding you. He’s also very focused on spaced repetition. If you buy into that, you end up teaching the portions of the field that are easy to structure in this way.
It’s useful and important to know those things. But if that’s the extent of your understanding of programming, you’ll be good at API recall and at the things that are easy to explain in atomic form. However, that alone isn’t sufficient for practicing as a software engineer or understanding the field deeply.
That’s not to say you shouldn’t do it. Execute Program as a product is fantastic. Learning those things is very useful. The question is, what’s the label on the front of the website? Does it claim, “Through this, you’ll understand all of it?”
Through a quantitative biology approach, are you going to become a biologist? Clearly not. But does it fill a gap in the typical way biology is taught? Yes, and there should be a course like that.
As an adult learner, it’s easier to say, “I’ll pay this much per month for this thing, and it’ll be part of my understanding.” Then you can supplement it with other things—watch a Grant Sanderson video, do the Math Academy calculus course, and mix and match what you need to learn.
That’s a good thing if we’re all clear with the student or user about what we’re providing and how it fits into the landscape.
Charlie: That’s all good stuff. Oz never watches sci-fi or anything, but I’m reminded of the 2009 Star Trek when they show Planet Vulcan and the learning pods.
Justin: Have you seen this, Justin?
Charlie: It shows Vulcan, and Spock as a young Vulcan is in a pod with a robot throwing questions at him.
Justin: Oh, wait, I know what you’re talking about.
Charlie: It sounds like Oz is suggesting Math Academy stays on that trajectory, which—
Oz: I don’t know the answer. It’s a really interesting question. How much could you go outside of what’s clearly working for Math Academy and make it work for more people with different motivations? Would that water it down? I don’t know. That’s a really interesting space.
Charlie: I’m curious to see how it goes. I have a slight tangent to ask about, Justin. Oz and I have been noodling ideas about self-publishing adventures over the past year or so. I noticed you have a bunch of textbooks that it looks like you’ve published. I’m curious about your thoughts on that. Are they self-published?
It’s quite awesome that you have a bunch of math textbooks. I’m curious about the story behind those and what your plans are with them.
Justin: Oh, yeah. It kind of started after college. I was writing just as a hobby. I thought it’d be cool to write a math textbook. I was doing a bunch of tutoring, and once you leave college, you feel your memory of this stuff starting to decay—especially if you studied something like math and then stop using it as intensely.
It gave me pleasure to write a textbook as though it was the one I wished I’d had when I was first learning the material. My understanding of that has evolved a lot, though, so I wouldn’t consider my earliest textbooks amazing pedagogically. Hopefully, they’re better than some others out there, but I wouldn’t call them the best, and they pale in comparison to anything we have on Math Academy.
Charlie: What’s your view on the textbook as a medium? A textbook is stamped in time, but you can have an online version where you push updates constantly. Do you still think a textbook is an effective medium for mathematics learning?
Justin: Honestly, no. I mean, the thing is, a textbook doesn’t adapt to the student. So much of learning efficiency comes from adapting every pedagogical decision to what the student is working on at that moment.
If we’re talking about breaking up a textbook into slides and presenting those at the right time, which is kind of what Math Academy does, then that sounds more interesting to me. But even if a textbook is easy to edit, I personally wouldn’t be motivated to work on it because I feel it’s far less efficient compared to Math Academy.
That said, if I were writing a textbook on something very niche, not covered by Math Academy—like for teaching a specific course or presenting content in my own way—then editability and ease of iteration would definitely factor in. Writing a textbook is very time-consuming, and provisional edits are important.
Charlie: Well, I’ve got to shuffle on and get back to things. Justin, this was amazing chatting with you. I’m really excited to dive in.
I’ve always wanted to do what Billy Madison does in the movie, where he goes back and starts kindergarten and works through each grade in a week. I’ve wanted to do that with Khan Academy, but it never stuck. I think it’s because I felt compelled to watch the videos.
I’m excited about Math Academy. I want to start from second or fourth grade, move through it, and win back some of those skill points I had in high school but have lost. I feel bad about it and want to get them back.
Justin: I hope you enjoy it. Once you get the wheels going, it’s just like working out—it might feel a little hard at first.
Charlie: For me, it’s about figuring out when during the day I’m going to do it.
Justin: Exactly. It’s just like working out.
Charlie: I might need to lay out my “math outfit” the night before, like people do with gym clothes. Or maybe go to bed with a calculator in my pocket or something.
Oz: What I do now is set math time. I’ve been doing this for a while. I sit down, and we’ve got Beast Academy for the kids and Math Academy for me. They do their problems, I do mine. That’s how I justify the time.
Charlie: Nice. I need to work on that. Okay, talk to you all later.
Oz: Great talking to you, Charlie.
Charlie: Bye, Justin.
Justin: Bye.
Prompt
The following prompt was used to generate this transcript.
You are a grammar cleaner. All you do is clean grammar, remove single filler words such as “yeah” and “like” and “so”, remove any phrases that are repeated consecutively verbatim, and make short paragraphs separated by empty lines. Do not change any word choice, or leave any information out. Do not summarize or change phrasing. Please clean the attached text. It should be almost exactly verbatim. Keep all the original phrasing. Do not censor.
I manually ran this on each segment of a couple thousand characters of text from the original transcript.
Want to get notified about new posts? Join the mailing list and follow on X/Twitter.