Transcript - Golden Nuggets Podcast #39 (Round 3): MA’s upcoming machine learning course

Rationale, vision, and progress on Math Academy's upcoming Machine Learning I course (and after that, Machine Learning II, and possibly a Machine Learning III). Design principles behind good math explanations (it all comes down to concrete numerical examples). Unproductive learning behaviors (and all the different categories: kids vs adults, good-faith vs bad-faith). How to get the most out of your learning tasks. Why I recommend NOT to take notes on Math Academy. What to try first before making a flashcard (which should be a last resort), and how we're planning to incorporate flashcard-style practice on math facts (not just times tables but also trig identities, derivative rules, etc). Using X/Twitter like a Twitch stream.

Cross-posted from here.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

Link to Podcast

Below is a smoothed version of the raw transcript.

James: Alright, delighted to be joined here once again by Justin Skycak, Chief Quant and Director of Analytics at Math Academy. So Justin, you’ve been working away on a cool new Matakami course. I’ve been seeing all the updates on Twitter. Wondering if you can tell us a little bit about that.

Justin: Oh yeah. Machine learning one. It’s going to be a legit machine learning course. We already have math for machine learning. So far, it’s like for anyone who signs up on our system wanting to learn machine learning, what they do is they go through this Foundations Sequence, Foundations One, Foundations Two, Foundations Three, that covers your standard high school on a bit of undergrad math, and then you jump into math for machine learning, which covers all the math that you’d need to know in order to take a proper machine learning course.

But up until now, we haven’t had a legit, like the actual machine learning course that goes through backprop and neural nets and that stuff. It’s all been just the math supporting that—like the multivariable chain rule, gradient, etc., probably working with probability distributions, whatever. But now we’re actually building out a course that covers these real algorithms, machine learning algorithms.

How the reason why we got started on this actually came out of a conversation with Jason Roberts, he and his wife Sandy, the founders of Math Academy. We were just talking, doing like a vibe check on how are our learners doing, what are they most excited about. I told them, “You know, Jason, I think seven, like if I were to estimate what portion of our adults on Twitter, X, are excited about machine learning, I would say that’s about 70% of everybody out there.” That would be my guess.

At this point, more than half of our users are people from Twitter, and they want to do machine learning, which makes sense. I mean, that’s kind of what you guys are interested in too, right? Right now it seems kind of silly that we support you with all the math up to machine learning, and then you get there, you know your math for machine learning, and then you’re like, “Okay, cool, I’m ready to go. Teach me machine learning,” and then we’re like, “Oh, just go find another course online.” Like, what?

No. And it’s the courses online—I have been pretty disappointed, honestly, with just all the machine learning resources out there. There are some halfway decent ones, but by and large, you know how you can pick up an algebra textbook or a calculus textbook, and it lays out topic by topic, concrete examples, here’s some nice problems, scaffolded? Now there are varying degrees of scaffolding. Some resources are better than others, but generally, you go to a machine learning book or resource, and it just kind of tells you in broad strokes, “How does the algorithm work?” and gives you some conceptual intuition about it so you can kind of check a box like, “Okay, I sort of understand this now.”

Then you run some code off a tutorial—there’s like, import TensorFlow, copy and paste code over, you write it, you get a result, and you’re like, “Wow, I did machine learning. That’s great.” But you’re not actually doing the math yourself. You’ve got to really get into the nuts and bolts of this stuff. There are some resources that kind of point you in that direction, but then you just hit a wall so quickly on the level of difficulty, and then you just kind of fall off.

This seems like the right area for us to build out more courses. So many people want it. There’s such a lack of good scaffolded resources. And it’s also kind of a missing piece in getting people to where they want to go career-wise. A lot of people sign up for Math Academy, like we’ve talked about in previous podcasts. You sign up, you want to skill up in math. The reason you’re doing that is because you think it’s going to transform your life in some way or improve your future, your prospects.

There is some amount of that that comes from just knowing math, being knowledgeable in math, but there’s a jump you’re trying to make. You’re trying to get yourself into a new position in life. For a lot of our learners, that’s what they’re trying to do. Right now, this knowledge of actual machine learning and ability to apply it is one missing piece we need to put in to finish that bridge to where you want to go.

That’s really interesting. That’s been the inspiration for it. We’re still going to do all these other math courses, like real analysis, proof-based linear algebra—we’re going to do everything. But the priority levels have shifted a bit. Previously, we were thinking, “Okay, let’s just finish all of undergrad math, and then expand out into math-adjacent subjects like machine learning and more computer science.” But right now, when the universe puts an opportunity in front of you and pushes you in that direction, it’s dumb not to take it.

So that’s the story behind the machine learning course. I should say this is going to be several courses. Initially, we were planning just one machine learning course, but then I started really scoping out, “What are the topics in this course going to be?” It turns out there’s a lot of topics in machine learning, as we all know. But when you start scoping it down to individual scaffolded topics, you start to realize, “Oh dang, this is more than a semester course.”

We talk about machine learning as if it’s just one standardized course, but that’s like talking about high school math as if it’s one course. No. You need to know arithmetic, algebra, geometry, algebra two, stuff in precalculus, calculus—all these subjects baked in. It’s the same with machine learning.

Machine Learning One will mostly cover classical machine learning—regression, clustering techniques, decision trees, neural nets—and stop around convolutional neural nets. If you pick up any classical machine learning textbook or look at any first-course syllabus, they top out around convolutional neural nets. They’ll introduce neural nets, backpropagation, simple multilayer perceptrons, feedforward nets, and then convolutional nets, which start to show interesting architectures.

This opens up a can of worms with topics to cover. You can’t do all classical machine learning and fit in all neural net architectures in the same course. That would go in Machine Learning Two. Machine Learning Two will be a second pass at topics in Machine Learning One, using more sophisticated training algorithms and exploring advanced setups, architectures, and model variations.

Most likely, Machine Learning Three will get into cutting-edge topics. For example, knowing transformers and doing some math with them puts you close to the cutting edge. But picking up a machine learning paper has even more sophisticated techniques, so that’s where the future courses will go.

Ultimately, we want learners to pick up a machine learning paper and know all the background information they need to read and implement it.

Justin: Well, I think the secret ingredient is just having experience doing this stuff in the past. On one hand, I usually focus on the technical aspects of the system as opposed to the content. But when I initially joined Math Academy, I was actually working as a content developer because that’s what was needed at the time, and I had a bunch of experience writing content. I used to do a ton of stuff—like hundreds of lessons in the system. I also spent three years teaching a highly accelerated quantitative coding course sequence within Math Academy’s original school program.

A lot of these topics, particularly in classical machine learning but also in coding in general, I spent three years teaching these to high schoolers who came in knowing calculus, multivariable, probability, statistics, etc. Something we always joke about at Math Academy is having to teach all this stuff to students. They had all the prerequisite knowledge because we taught it to them. We taught them the foundational math behind the stuff.

It can be more challenging to get something to work for a kid with a lower attention span and, more generally, more adversarial learners. Especially in a school program, it’s not always 100% intrinsic interest. If you can make it work for kids, then you can do it at scale for adults. It comes down to having years of experience doing all this manually, which is hard to replicate without that experience.

Me and Alex have tried using ChatGPT in the content development flow. It can be good for idea generation sometimes, but it doesn’t solve the problem for you. At the end of the day, you need a good mental model of how a student thinks about these things, what confuses them, and what it takes to scaffold concepts. It also helps to have done this for math subjects like algebra, calculus, and differential equations, where it’s more straightforward to scaffold concepts.

With math courses, you can flip open a textbook and find scaffolded explanations. It’s not always at the ideal granularity, but it’s at least directionally correct. In machine learning, many resources I’ve seen online aren’t even directionally correct. Knowing how to scaffold math has helped us scaffold machine learning, as math provides a simpler case for learning how to do this effectively.

James: There’s this phenomenon in math, where sometimes you read an explanation, and it’s so good you can’t help but understand it. Even after reading it, you couldn’t make yourself not understand it. Sometimes you just nail the explanation perfectly. Have you found any design principles for coming up with explanations that achieve that level?

Justin: I think there are a couple of components to it. There are two groups of principles. One group is inherent to the explanation itself. The other is about being up to speed on your prerequisite knowledge—not just having learned it but having it relatively fresh in your head. If you learned something five years ago, forgot it, and think you’ve learned it, that’s not enough. You need to be spun up on your prerequisites.

When that condition is satisfied, a good explanation can help things click. The first thing that pops into my head is having a concrete example. Drawing analogies and illustrating concepts conceptually is helpful, but a concrete example with actual numbers and a simple case you can wrestle with hands-on is critical.

In machine learning, concepts like overfitting, underfitting, bias, and variance boil down to numerical measurements on functions and data. A concrete example acts like a ledge you can hold onto. Anytime you feel confused, you can go back to that example and ground yourself.

Math explanations can be hard to follow when they’re too hand-wavy or nebulous. You might think you’re following them but later realize you’re confused because the explanation didn’t stick. Concrete examples make you immune to this kind of confusion.

Zander: Is there a way you avoid common errors in explanations, like the curse of knowledge? Sometimes explanations make sense to the writer but not to someone fresh. Do you rely on experience, or is there a formal process you follow?

Justin: There are a couple of ways. Experience definitely helps. For example, I know from experience that students often mix up certain things, so I clarify those differences in the explanation.

We also run analytics on all the lessons we put out. We look at the percentage of students who pass on the first try, within two tries, and where people get stuck. We can drill down to specific knowledge points or questions within a lesson to identify issues. If needed, we scaffold more in that area or split topics into smaller pieces. We’ve been doing this for years, so we have a good idea of common pitfalls and how to avoid them.

The main thing is still concrete examples. Writing a concrete example forces you to address potential misunderstandings. When you write a solution, you see the steps and can identify gaps or areas where prerequisites are missing. It also helps to compress explanations. If your explanation is too long, it means you’re trying to do too much. Concrete examples make it easier to identify when to offload cognitive load into lower-level topics.

Zander: How do you balance all that with the one XP per minute idea? Does that affect how you write questions, or is it more about adjusting the number of questions or XP?

Justin: The XP is done in hindsight. We don’t write lessons to match a specific XP number. Instead, we write the lesson to be bite-sized and manageable. If we do that, the lesson typically ends up being between 7 and 25 XP. Some topics are naturally more complex and take longer, while others are simpler and quicker.

We calculate XP based on how many knowledge points, tutorial slides, and questions are in the lesson. We estimate question times with input from content writers, who average their estimates. Then we calibrate these estimates with real student data to account for actual time spent. It’s a manual process that gets refined with data.

Zander: Does response time on questions play a role in spaced repetition?

Justin: Yes, response time can play a role. It helps gauge difficulty and adjust the spacing of repetitions to optimize learning. However, the details of how that works depend on the system and algorithms we use.

Justin: For diagnostics, time does matter. For lessons, reviews, and other tasks, it doesn’t. For quizzes, it sort of does. Let me explain.

For diagnostics, we measure time. If you take a long time to solve a problem, we won’t give you much credit for solving it. We’ll give you a little, but we don’t want to place you too far ahead of your ability. For example, if someone takes five minutes to solve a quadratic equation, they’re not ready to move far beyond that. They need more practice.

Zander: And that’s a direct result of the automaticity built into the system. That’s the goal, right?

Justin: Right. The goal is to get you to the point where you can do foundational skills quickly without using much mental effort, so you can focus on learning the next thing. For that reason, people sometimes get frustrated with the diagnostic and place lower than they expect. They might say, “I learned topics XYZ in school,” but the question is whether they can solve problems correctly, consistently, and quickly. If they can’t, they need more practice. It doesn’t matter if they covered it in school; we’re measuring whether they’re ready to build on that skill.

For lessons, right now, only your accuracy matters in spaced repetition mechanics. If you solve questions correctly, you get a lot of positive credit. If you miss questions, you get less. Failing tasks gives you negative spaced repetition credit. There’s more variance in lesson performance, though. Students may come out of a lesson with different speeds, but if a student solves questions correctly, even slowly, we don’t want to hold them back by making them repeat the lesson excessively. They can implicitly practice those skills by continuing with advanced topics.

However, if a student is slow to solve problems, we might trigger reviews earlier to help them get faster. As we integrate task behavior analysis, like whether a student constantly goes back to examples or reference material for every problem, we’ll incorporate that into the spaced repetition mechanics. For instance, if someone relies too heavily on examples, they’re not learning effectively. We want to incentivize proper learning behaviors. If a student consistently tries questions without examples and builds skills, they’ll earn more XP as a reward.

James: Since you’re the Chief of Analytics, you’ve probably looked at a lot of student behavior data. Have you seen anything mysterious or interesting in student behaviors—something unexplained?

Justin: That’s a good question. This reminds me of something from the early days. I noticed kids doing well on quizzes but then getting follow-up reviews on questions they got correct. At first, I thought the model was broken. When I dug in, it looked like the database was changing itself. I was baffled. I talked to Jason, and he immediately said, “That’s students cheating.” He was right.

We investigated and found students were exploiting the system. They’d open the quiz in a new tab, see what they got wrong, figure out the correct answer, and then change it before submitting. It was an exploit we later patched. Anytime we see weird behavior, it often turns out to be someone trying to game the system.

Now, if your question is more about unusual learner behaviors that seem productive but aren’t, I’d say it’s not so much mysterious as it is predictable at this point. For example, some students will guess all the answers in a diagnostic because they don’t want to take it. Others will spend half an hour solving one question, looking up how to do it online, and then complain the diagnostic took five hours.

Some skip tutorial slides and examples, try to solve a problem, fail, and spend ten minutes confused without going back to the material. I saw this a lot when teaching classes. On the first day with Math Academy, there’s always that one kid who races through without reading anything and gets stuck.

Other students fail tasks repeatedly but don’t stop to understand why. They don’t read solutions, make the same mistake over and over, and expect different results. These behaviors are common, especially with kids. They’ll often fall off the rails unless you guide and coach them.

Sometimes, we even get emails from parents saying their kid is struggling, but when we check the data, the kid spent two seconds on a question, didn’t read the example, and skipped everything. In some cases, kids genuinely try but forget there’s a tutorial slide or example available. Other times, they’re intentionally gaming the system or trying to avoid work.

One time, a parent claimed their kid worked four hours daily for a month but couldn’t gain XP. When we checked, the kid spent eight minutes total over two weeks. It’s a mix—some behaviors are unintentional, others in bad faith.

Zander: If an adult were using Math Academy, what would be their incentive to try trickery like that? Isn’t it all self-directed?

Justin: The adults are typically less adversarial. One failure mode adults sometimes get into, especially during diagnostics, is grinding through questions far beyond their capabilities. The reasoning is often, “I just want to get as far as I can and face the most challenging problems. I’m supposed to try as hard as I can on this problem, right?”

In reality, the purpose of the diagnostic is to determine whether you can solve problems comfortably, quickly, and correctly—or if you need more practice. What they should do is say, “Okay, I don’t know how to do this without reference material,” or, “I covered this five years ago, but it’d take me half an hour to figure it out again.” Just select “I don’t know.” There’s no need to struggle unnecessarily. You’ll get more practice as you go.

Don’t try to fake the system into thinking you know how to do the problem because that will place you further ahead than you should be. This behavior is often in good faith—they’re trying to work as hard as they can and put their best foot forward—but it’s not the best approach.

We need to handle this better in the diagnostic. Even though we put instructions on the screen, many people don’t read them. It might help to include a trigger that pops up if someone is taking too long on a question: “Hey, it seems like you’re taking a while. If you’re not sure how to do this, just click ‘I don’t know.’ Don’t grind through it.” After a while, the system could automatically move them to the next question.

Zander: That’s good practical advice. If you’re listening and haven’t done the diagnostic yet, remember: you’re trying to be automatic on everything. Don’t spend an hour figuring something out or looking it up. You need to recall it and do it with ease. That’s straightforward advice.

On another note, are there any good habits that some learners apply that more people should adopt? For example, quizzes are meant to be purely recall. If you have to look something up, you didn’t get the quiz answer. I imagine looking things up during quizzes can mess with task scheduling. Is that correct?

Justin: That’s absolutely correct. Quizzes are closed-book. They’re designed to gauge whether you need more practice. If you have to look something up, it means you don’t have the recall we’re aiming for.

I’d rank that as the second most important suggestion. The number one suggestion is to avoid relying too much on examples or solutions from previous problems. Initially, use the reference material to figure out how to solve a problem. But if you’ve gone through a worked example and feel ready, try to solve the problem on your own. If you forget how to start, go back to the example briefly, but don’t keep it open in a separate tab for every problem.

The goal is to recall as much as possible. Think of it like using a spotter at the gym. You’re the one lifting the weight. The spotter only helps if you’re really struggling. If you rely on the spotter for everything, you’re not building strength. Similarly, overusing examples prevents you from engaging in retrieval practice, which is essential for learning.

Some adults, in good faith, keep examples open for every problem. They think they’re being thorough, but they’re not practicing recall. Others take detailed notes and use them as a crutch during reviews, which undermines learning.

Zander: The goal is to activate the testing effect by trying, right? What if someone keeps trying but still has to rely on the example every time? Should they stop looking at the example and just give the wrong answer?

Justin: The goal is to try to recall every time. The testing effect is about successful retrieval, but consistent effort to reduce reliance on examples helps you get there. If you rely on the example less over time, that’s progress.

If you repeatedly can’t start problems and always need the example, it may indicate you’re not trying hard enough to remember. It’s like going to the gym but never increasing your weight or reps. If your performance isn’t improving, something’s wrong.

Zander: What about skipping the example for introductory problems in a new knowledge point? Is there value in trying to solve it first, given that techniques often build on earlier points?

Justin: There’s a tradeoff. The safe approach is to review the example first, but skipping it can work if done carefully.

The risks include spending too much time on a problem, which has a high opportunity cost. You could finish the lesson in that time. Another risk is solving a problem with an overfitted technique that works for one problem but fails on others. This can lead to frustration and wasted time.

Even if you solve the problem, it’s good to review the example afterward to compare your approach to the suggested one. Practicing a flawed technique builds bad habits, which are harder to fix later. You don’t want to build automaticity on the wrong thing.

Zander: That’s similar to sports, where athletes develop bad habits. For example, a fighter tilting their head during an uppercut puts themselves in danger. It’s hard to retrain those reflexes.

Justin: Exactly. Building automaticity on the wrong behavior creates reflexes that are difficult to unlearn. Catching those issues early and sticking to proper techniques is crucial.

Zander: That makes a lot of sense to me. And just so people know, struggling with it is not a bad thing from a general learning perspective. If you struggle with it for three minutes and then are able to do it successfully, that’s actually a very positive comment on your future ability for that task. In the future, you’ll probably be able to do it better as a result of having struggled. That’s just from a general learning perspective.

Justin: Totally agree. That’s one of the main features of spaced repetition, right? You let your memory decay to a point where it’s difficult to retrieve the information. But if you overcome that difficulty, it significantly increases your retention. Desirable difficulties in general.

Zander: Also, don’t be neurotic about it. If you have to look at the review, it’s fine because there are closed-book tests and quizzes where there’s no review. If you get it wrong, you get it wrong. The way Math Academy works, higher-level problems will eventually integrate that material without explicitly mentioning it. If you don’t fully understand something, you won’t be able to get those problems right, and it will be reintroduced in the review process. If I’m understanding correctly, the areas where you’re misunderstanding will come back in the review process, and then you’ll be good to go. So don’t be neurotic. Just try to get the most out of it.

Justin: What you’re describing, we typically refer to as layering. You keep building additional knowledge onto what you’ve learned. It gives you implicit practice with lower-level skills. The more ingrained those lower-level skills become, the stronger your overall knowledge base. It’s like coding a project where you build new features or capabilities, and often you realize you need to refactor lower-level components. As you build more, those lower-level components become really strong. It’s the same with knowledge.

Zander: That’s one of the main benefits of the dependency structure. It introduces and layers everything so smoothly. You’re using a skill you acquired last week in a natural way. Instead of just solving a set of problems about that skill, you’re implicitly using it, and it becomes much more natural to do something that was previously a deliberate learning process. It adds so much integrity from a neural perspective, generalizing the connections and making the skill more stable. That’s a huge benefit of the system overall.

But I have one more point about getting the most out of it. I think I’ve seen you mention on Twitter that you don’t recommend taking notes. Some people make flashcards, like Anki flashcards or definitions. Do you explicitly recommend against making notes?

Justin: Yes, I explicitly recommend against taking notes. There’s a difference between taking notes you’ll refer to in the future and using as a crutch versus just thinking on paper or diagramming something out. The reason I say don’t take notes is that it’s too tempting to use them as a crutch. You want to make looking up material slightly annoying. If your notebook is right beside you, the temptation to flip to the answer will be high, and you’ll rely on it too much.

Especially if you’ve put a lot of effort into your notes, you’ll feel compelled to use them, which undermines the retrieval process. You need to try retrieving as much as possible without reference material. Reference material should be a last resort. If you need it, you can brush up on a topic in the system. It might be slightly annoying, but that’s a good thing—it discourages over-reliance.

If you’re reading an explanation or working something out on paper, diagramming concepts, or thinking on scratch paper, that’s fine. Those notes are temporary and won’t become a crutch. The pitfalls to avoid are relying on notes as a crutch and spending too much time creating them, which can slow your learning process.

Zander: The time issue is important because it can be demotivating. If it takes five minutes to get one XP instead of one minute, it’s inefficient. Also, notes are more useful when the material’s specificity doesn’t match your learning level. For example, reading a 300-page book where only five pages are relevant makes note-taking valuable. But in a system like Math Academy, every knowledge point is useful in a particular way. You’d be taking notes constantly, and the system already organizes information accessibly.

Justin: Exactly. The system is intentionally designed to provide quick access to reference material if needed, but not so easy that it becomes a crutch. We try to strike a balance, and you don’t need to take notes on paper to refer back to.

Regarding flashcards, some people ask if they need to make flashcards for derivative rules or similar topics. The answer is both yes and no. Don’t make flashcards for everything you read because the spaced repetition system will handle most, if not all, of your review. It’s normal to forget some material early in spaced repetition because your memory decay curve is steep. With time, as you progress further in the spaced repetition process, this issue diminishes.

If, after several weeks and multiple exposures, you still struggle to recall certain facts, then it might make sense to create flashcards. But first, ensure you’re engaging in proper retrieval practice. Ask yourself: Are you trying to recall the material, or are you defaulting to looking it up or relying on notes? If you’ve ruled out those issues, then making flashcards might help, but that shouldn’t happen often.

This is something Jason and I have discussed incorporating into the system. When someone struggles with things like trigonometric identities or derivative rules, these are essentially math facts—an extension of multiplication tables. We want to build timed, drill-style practice for these facts into the system, similar to how kids learn multiplication tables. It’s a feature we started working on but had to pause for other priorities.

Zander: That makes sense. Flashcards could also help with recognition-type problems, like remembering the names of shapes or definitions of terms.

Justin: Exactly, for recognition-type problems or instances where solving isn’t the issue, but recalling a specific term is. That’s where flashcards might be helpful.

James: Exactly. This is kind of tangential but related to the analytics discussion. I’m curious—how much more information would you like to gather about the user? Say, in a science fiction ideal world where everyone has a neural link, and you could read and write data from their neural link API, what additional information would you want beyond what you already have in the system to make the educational experience better?

Justin: That’s a really interesting question. It’d be amazing if we could know exactly the neural connectivity of all these math topics in students’ brains. We have this knowledge graph of math topics and how far along you are in the spaced repetition process, which is an estimate of how solidified this knowledge is in your brain. But these are behavioral estimates based on answering questions correctly or incorrectly.

It’d be incredible to know the biological health of these connections, though that seems far off. When I was in college, I got interested in computational neuroscience. Coming from a math background, I thought, “Wouldn’t it be amazing to have a dataset of neural connectivity and all the properties of neurons?” The current level of granularity we get from reading brain signals, even with brain-computer interfaces, is still very aggregate. You can apply machine learning to match these metrics to specific actions, but understanding the conceptual mapping of information in someone’s brain seems very far away.

It would be incredible to have a physical MRI of the student’s brain. Our knowledge graph is like an MRI—it’s an approximation of math connectivity. If or when this technology exists, there will undoubtedly be ethical issues to address. It’s an interesting question, but for now, it’s very speculative.

James: I was interested in brain-computer interfaces at one point, but as you said, the signal is so noisy that it’s unlikely you can derive anything truly useful, which is unfortunate. Maybe not brainwave reading, but could monitoring more physical signals like eye tracking, facial expressions, or yawning help? For example, I’d love a reading app that detects when I’m bored and immediately switches to a new article, keeping me engaged—like a TikTok-style algorithm for reading.

You can already infer boredom with behavioral metrics, like scrolling speed or rapid clicks. For instance, when I’m bored, I rapidly click around the answer button.

Justin: That’s interesting. In the ideal learning system, it would respond to emotional or cognitive states, like a good coach who notices when an athlete isn’t in the right headspace and adjusts the exercise.

For now, we start by gathering as much information as possible from student clicks, navigation, and behavior within the system. You can infer actionable emotions, like frustration or boredom, from these behaviors. But introducing more affective measures—like tracking physical signals—has trade-offs. There are privacy concerns, especially with kids, and it complicates the system. We typically avoid collecting sensitive data that might make people uneasy. Maybe far down the road, if it significantly improves the system, it could be worth the headache, but it’s not something we’re pursuing now.

Zander: Imagine registering for an app that asks for your neural link data to optimize your learning. That’d be insane. But as you said, we’d also need research to interpret the data and decide what to do with it. For example, sports coaches might put activity trackers on athletes, but what are they doing with that data? If a tracker shows a player took 10,000 steps in practice, does it improve performance, or is it just another number?

When we get neural data, people will need to figure out what it means, how it correlates with real life, and what interventions can improve it. It’s many steps away.

Justin: Exactly. You’d need insights into what’s happening in someone’s brain that result in actionable changes, different from what you could infer with behavioral data. Otherwise, you’re just measuring metrics to reach the same decisions a coach could make by watching a player on the field. Of course, they just need to run more. If they only ran three steps in practice, you don’t need a neural link to figure that out. It’d be interesting to see how this technology develops, though.

Zander: About the knowledge graph: I’ve seen people sign up for “Mathematics for Machine Learning” as their course. Is it true that no matter what course you choose, it fills in whatever is needed to reach that point in the graph?

Justin: Pretty much. There is a limit to how far it looks back, at least for now. It’s on my to-do list to refactor the diagnostic algorithms to look all the way back.

For now, “Mathematics for Machine Learning” looks back to early high school math. If you can reasonably take the course but have foundational gaps, it will fill those in. However, if you don’t know how to add fractions and sign up for Multivariable Calculus, it probably won’t look back that far.

If you’re remotely confident that “Mathematics for Machine Learning” is appropriate for you, we can likely capture all your missing foundational knowledge in the diagnostic.

Zander: If you finish that course and then go back to Foundations 2, will you be 100% on that course?

Justin: Not necessarily. When you take a course diagnostic, we assess your knowledge of that course and its prerequisites. For example, someone taking Calculus would be assessed on Precalculus, Algebra 2, and Geometry but not on topics like matrices, which are common in Precalculus but not in single-variable Calculus.

If you completed the Calculus course, there might still be gaps in Precalculus or Geometry that weren’t assessed. You could always go back to fill in those courses. Our goal is to get you to complete the topics in your chosen course as efficiently as possible.

Zander: When you do the diagnostic and complete Foundations 1, and then move on to Foundations 2, I’ve seen some people start at 0% or close to it. Why is that? Wouldn’t the diagnostic transfer some progress, especially if there’s overlap between the two?

Justin: If you just take a diagnostic for Foundations 1, it doesn’t assess anything in Foundations 2 unless there’s overlap between the two. Sometimes courses have topics in common, so that’s possible.

Another scenario is that someone might take a diagnostic for Foundations 2 or 3, struggle with it, drop down to Foundations 1, and then start climbing back up. They may have gotten credit for some higher-level topics based on the earlier diagnostic, which happens fairly often when people drop down.

Zander: That makes sense. Here’s a different question: once you finish something like Mathematics for Machine Learning, do you still need to do reviews every day? How much time does it take to maintain that knowledge?

Justin: Yes, you’ll continue to do reviews. If you finish all the courses in your sequence, you unlock an Easter egg in the system. It starts feeding you topics from other courses, including unfinished ones. For example, after completing Mathematics for Machine Learning, you might start seeing topics from differential equations, abstract algebra, or probability and statistics. These come from unfinished courses but are well-developed enough to provide review material.

If you just want to maintain what you’ve learned, say for a future course, you can put yourself into test prep mode. This keeps you in the current course and continues feeding you reviews. Maintaining knowledge would probably take around 15 minutes a day or about an hour a week, though that’s just an estimate.

Zander: I imagine that intuition about spaced repetition doesn’t fully apply here because of the dependency graph. If you’re using earlier knowledge frequently in later lessons, wouldn’t your probability of recall be much higher, even after a long time?

Justin: Exactly. If a low-level topic comes up frequently in high-level topics, you might never see an explicit review for it. Instead, you’re implicitly reviewing it while working on related high-level tasks.

Every task we assign is optimized to elevate your entire knowledge profile. The system computes which topic review will provide the most benefit while covering everything due. It’s a highly optimized process, ensuring reviews are efficient and effective.

Sometimes people don’t realize this and wonder why they never saw a review for a specific topic. It’s because reviews are happening implicitly. We’ve considered adding visualizations to show how credit trickles down the graph—something like an animation showing how one review extends the spacing for related topics. It would help people understand what’s happening behind the scenes.

Zander: Visualizations like that would be great. Even after a lesson, showing where it fits in the graph would be helpful.

James: A skill tree visualization would be awesome. It’s interesting about the Easter egg. Showing previews of unreleased material might motivate people to complete courses faster.

Justin: That’s an interesting idea. The Easter egg depends on the course being connected to the graph. For example, content from our original school program is connected, so you might see topics from university-level courses. But the Machine Learning course is still in development and isn’t fully connected. Even if live topics exist, they might not appear as part of the Easter egg. It’s mostly for university-level courses we’ve already developed but haven’t officially released.

James: You could tease that on Twitter. It might generate excitement.

Justin: That’s a good idea. The challenge with connecting courses to the graph is ensuring the structure makes logical sense. If you’re heavily editing a course, it gets tricky because everything must be validated carefully. It slows down the process, but it’s worth considering.

Zander: I hate to ask, but do you have a timeline for when the Machine Learning course will be ready?

Justin: Don’t hold me or Alex to this, but our goal is the end of February. It’s all hands on deck for this course. I’ll be working on it full-time, along with Alex, Yuri, and a couple of other content writers. Progress is good so far, and it feels achievable if everything goes to plan.

Zander: How much new material do you need to learn for this? Or is it mostly brushing up?

Justin: For classical machine learning, I already have a solid understanding. I’ve taught these topics in Math Academy’s school program for years and even wrote a textbook on it. Most of the work is about scaffolding examples precisely and revisiting core concepts to ensure clarity. It’s less about learning new material and more about refining and presenting it effectively.

Scoping things out to the level of granularity we need is honestly pretty fun for me. It’s right at the edge of difficulty—not too hard if you have the background knowledge, but sometimes the background knowledge has faded a bit or isn’t as sharp as you’d like. It’s satisfying to fill in those little gaps.

For me, it’s less about relearning how something works and more about figuring out how to scaffold it effectively. Creating examples that can be done by hand in a couple of minutes, rather than taking 10 or 15 minutes, is a key challenge. We also want to design problems that allow variations so students can practice the same type of problem multiple times to drill the skill.

That’s difficult because machine learning resources often don’t scaffold concepts the way algebra or calculus materials do. Most of the effort goes into figuring out how to scaffold these concepts well, but it’s coming along nicely.

Zander: I imagine there are points where the learning only works if you run some code or do something hands-on. Is that the case, or are you keeping everything within the platform?

Justin: For the topics in the course, the lessons will focus on the math by hand, similar to the linear algebra course. Initially, Jason and I thought the course might need students to write code, but the core skills are more about understanding the underlying math than the coding itself.

We do plan to include mini-projects where students can apply the math to write code. There are ways to keep this within the platform, like having a Python editor where students write code that outputs specific results. It’s similar to how our free-response questions work. Students input a symbolic expression, and we evaluate it for random inputs to confirm correctness. We could apply the same concept to code.

Zander: That’s surprising. I assumed there would be more coding.

Justin: Not as much as you’d think. A lot of machine learning tutorials and classes are code-heavy because they focus on teaching frameworks like TensorFlow. That’s useful for a machine learning job but different from understanding the math behind it.

By focusing on the math, we can turn concepts into manageable problems, like practicing one or two iterations of gradient descent rather than doing 100 by hand. For example, we might ask about stopping criteria after a thousand iterations, the slope of the loss function, or the absolute difference between estimates. This lets students practice parts of the algorithm without executing the whole thing. A coding project could then bring it all together.

Zander: That approach makes sense. It seems like a solid way to balance things.

Justin: Definitely.

Zander: We’re coming up on two hours. I wanted to ask how you personally learn, especially since there’s no Math Academy course for new topics you’re tackling. But I’m short on time.

James: We’d be happy to do another podcast. There’s always more to discuss.

Justin: I’d be happy to join again. It’s always fun talking with you. You ask such interesting questions. On Twitter, I often get the same questions repeatedly, but with you, it’s new and thought-provoking.

Justin: You also give great concrete examples, like the uppercut analogy. That was fascinating.

Justin: Yeah, that was a great example. I’d love to chat again.

Zander: Next time, we can dive into how you learn and manage productivity. You seem to do a million things, and I’d love to learn from that.

James: Agreed. You’re impressive on Twitter as well. It’s a bit intimidating.

Justin: About posting—initially, I’d carefully craft one perfect tweet a day. Then Jason told me to treat Twitter like a Twitch stream: share whatever you’re working on. At first, I worried posting too much would hurt visibility, but it seems the opposite is true. More posts mean more chances for something to resonate.

Even if your language isn’t polished, people engage with interesting ideas. Now, my strategy is more stream-of-consciousness, as long as the content has some value. I don’t want to go too far and post anything random, but I’ve found a good balance.

Zander: That sounds like a great approach. It’s impressive how much you accomplish. I’d love to hear more about your methods and learning strategies.

Justin: I’d be happy to discuss it next time. Thanks again—it’s been a lot of fun.

Prompt

The following prompt was used to generate this transcript.

You are a grammar cleaner. All you do is clean grammar, remove single filler words such as “yeah” and “like” and “so”, remove any phrases that are repeated consecutively verbatim, and make short paragraphs separated by empty lines. Do not change any word choice, or leave any information out. Do not summarize or change phrasing. Please clean the attached text. It should be almost exactly verbatim. Keep all the original phrasing.

I manually ran this on each segment of a couple thousand characters of text from the original transcript.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.