Watch on YouTube or Listen on Spotify

What we covered:
– A lot of schools have recently begun using Math Academy in their classrooms. And one of the biggest benefits of using Math Academy is that it automates all the mechanical parts of teaching, like writing questions, keeping track of what students know and what they don't know, monitoring student progress, assigning extra practice when needed, grading, all that grindy stuff.
– None of these tasks is enjoyable. They suck. Just ask any teacher. I mean, we grinded through all that back when we were teaching ourselves, and it takes so much effort just to get even a halfway decent approximation of doing it right. And there's just a limit to how well that you can do it if you're doing it manually. It's the whole reason why we built the system.
– And what that system does, what Math Academy we does is it frees up teacher bandwidth to focus on the human elements of teaching: building relationships, connecting what students are working on to their own unique interests. Those kind of things that enhance the learning experience, but that really can't replace skills practice.
– I mean, in-class projects can be great, but only if students have the prerequisite knowledge to be successful with them. If they don't, then projects are frustrating, and the students who understand the material will end up doing all the work and carrying everybody else, who will learn next to nothing. It's inefficient and frustrating all around unless students have their skills in place.
– Ultimately, if students don't master the math in each class, they'll be unprepared for the next one. And in a subject as hierarchical as math, these gaps compound quickly. True empowerment isn't simply telling students they have potential. It's making sure they actually have the real skills to move forward and realize that potential.

0:00 - Introduction
2:56 - What is the teacher’s role alongside Math Academy?
5:37 - Math Academy frees up teachers to do the human parts of teaching
7:03 - Projects are great if students have the prerequisite skills
7:42 - Drills without context are boring
8:43 - Games without skills are inefficient
11:14 - Build fun activities on top of a solid foundation of skills
12:15 - Teachers can tailor the class to the students’ preferences
13:28 - Implementing mastery learning is too much work for a single teacher
15:27 - Doing projects without prerequisites is frustrating
16:57 - True empowerment is giving kids the skills they need to succeed
19:30 - Missing skills compound in hierarchical skill trees
24:06 - Lack of automaticity in lower level skills slows down higher level tasks
27:14 - The MA team builds and improves courses through experience
29:21 - The MA team targets tasks with low pass rates for additional scaffolding
31:03 - Alex built knowledge graph intuition through years of experience
37:40 - Social media enforces hyper-accountability
39:19 - Differential equations courses are often a hodgepodge of disjointed techniques
43:20 - Math Academy university courses are a superset of elite university content
45:18 - Differential equations is a highly branching subject
49:21 - The breadth of Differential Equations makes it often poorly taught

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

Watch on YouTube or Listen on Spotify

The raw transcript is provided below. Please understand that there may be typos.

Justin Skycak (00:00) Welcome to the Math Academy podcast. I’m Justin Skycak, Chief Quant and Director of Analytics at Math Academy, and I’m here with our founder, Jason Roberts, and our Director of Curriculum, Alex Smith, to discuss the role of teachers alongside the Math Academy system.

A lot of schools have recently begun using Math Academy in their classrooms. And one of the biggest benefits of using Math Academy is that it automates all the mechanical parts of teaching, like writing questions, keeping track of what students know and what they don’t know, monitoring student progress, assigning extra practice when needed, grading, all that grindy stuff. None of these tasks is enjoyable. They suck. Just ask any teacher.

I mean, we grinded through all that back when we were teaching ourselves, and it takes so much effort just to get even a halfway decent approximation of doing it right. And there’s just a limit to how well that you can do it if you’re doing it manually.

It’s the whole reason why we built the system.

And what that system does, what Math Academy we does is it frees up teacher bandwidth to focus on the human elements of teaching: building relationships, connecting what students are working on to their own unique interests. Those kind of things that enhance the learning experience, but that really can’t replace skills practice.

I mean, in-class projects can be great, but only if students have the prerequisite knowledge to

be successful with them.

If they don’t, then projects are frustrating, and the students who understand the material

will end up doing all the work and carrying everybody else, who will learn next to nothing.

It’s inefficient and frustrating all around unless students have their skills in place.

Ultimately, if students don’t master the math in each class, they’ll be unprepared for the next one. And in a subject as hierarchical as math, these gaps compound quickly. True empowerment isn’t simply telling students they have potential. It’s making sure they actually have the real skills to move forward and realize that potential.

So in this episode, we’ll talk about how teachers can build on the foundation that Math Academy provides to create rich learning experiences in the classroom.

Justin Skycak (02:01) So there’s two topics that we had on the docket last time that we didn’t get to. here they are. So I figure we could go through them today. So first of all, how can you get the most out of using Math Academy in conjunction with a teacher or tutor?

Second of all, failure modes. What are some of the most common errors in teaching or in self-studying? So I don’t know if you guys like one of those over the other, or want to jump into one of those first.

Jason Roberts (02:32) let’s start with the first. Because I get this question a lot when I’m speaking to educators. It’s like, you know, I’ll talk to some principal or head of school or head of academics or something and they’ll be like, okay, we’re really interested in Math Academy. It’s just a lot of problems, really exciting. You know, we have kids at different levels, who have different needs and we’re trying to, Math Academy can help us address that, et cetera, et cetera. But how do I use this on classroom? Like, what does the teacher do?

Justin Skycak (02:32) Yeah. Yeah.

Yeah, yeah, I figured that’d probably be good of him.

Jason Roberts (02:59) I’m like, okay, here’s how this works. this school where we, Math Academy came from the Pasadena Fight School District where I used to teach some classes and Justin used to teach classes. And this is where a lot of these ideas were sort of given a trial run and we’re, you I was just experimenting with stuff back, know, whatever. Yeah, Crown Zero says, I don’t know, this is 2014, 15, kind of thing,

Justin Skycak (03:16) Yeah, it’s ground zero for everything.

Jason Roberts (03:25) And then eventually when it became more of automated system, you know, we hired some PhD instructors to teach with the middle school and the high school classes, because once the kids got to high school, they were doing, you know, university level courses. Okay. So we had these guys come in and we had this new automated system and they’re like, they had the same question. Okay. So what, am I supposed to do? I’m like, all right, here’s what I would recommend. Use the first 10, 15, 20 minutes of class to do some kind of group oriented activity. It could be.

Hey, you know, we’re gonna talk about Pythagorean theorem. Let’s talk about Pythagoras. What was math like back in ancient Greece? what was, you know, like was some weird group of people and they had a whole, like a cult, math cult. I you you can talk about full stuff, right? Or, hey, I got two, or I got this really great challenge problem, ready for the Pythagorean theorem. We’re gonna work on this. I’m gonna advise you up in the teams of two or three. You know, I’m gonna give a, you know, maybe I’ll give a hint, you know, something like that. So we got a challenge problem.

Right? Or you might say, okay, we got some ongoing projects. You know, so we got this project where you guys are gonna use a Pythagorean theorem to construct some things, some buildings, some space structure, whatever you came up with, some cool thing that the kids can do, can apply the Pythagorean theorem. So you spend the first 15 to 20 minutes doing something like that. It’s a little more social, more engaging, which is a lot more interesting than normal math class where you just sit there.

I just take notes with a teacher talks, right? You do stuff. And then after that, you say, okay, you can use the rest of your time to use the Math Academy system. So you can open your laptops, work on that. And that way you get a solid 30 to 40 minutes. And then you got like zero to 10 or 15 minutes, maybe 20 minutes tops homework at night. That’s it for math. And when you’re talking like even, I mean, I had that much at least even when I was in eighth grade, which is high school, we were talking.

Justin Skycak (05:19) Yeah, same.

Jason Roberts (05:19) 30 minutes

an hour, pre-calculus, algebra two, calculus, checking an hour, homework a night. So they get that work done in class and like that, they not only do not have a lot of homework, so they’re much happier with the class. Like, oh, it doesn’t burn my night away every night, right? It’s like, a lot, stressful, but you can walk around as an instructor and help them, right? You can be like, Sarah, I saw that you struggled on that one topic yesterday. Why don’t you come over here? Let’s just talk through a little bit.

You know, she might think, ⁓ I get out there. Okay, okay. Well, Joseph, like what happened with, you know, this? He’s like, okay, let’s look through it. And you sit down with them and you see, remember, you gotta be careful with your negative signs. You write that stuff down, right? Like, okay, I get it. So you walk around, encourage them, you know.

Justin Skycak (06:07) It just frees you

up to be a human, basically, right? Like let the robot be a robot and do the robotic things and that way you can be a human. it’s like the problem with a lot of math classes is that the instructor tries to do all this human stuff, ⁓ you know, like cool projects and stuff, doesn’t actually do the the grindy skill portion, right? And that’s like, it’s like basically the equivalent of like, okay, everybody, let’s do a slam dunk competition. Meanwhile, nobody can even touch the rim.

Like nobody even has like any level of hops. It’s like, okay, well, how about we let the system get you your vertical jump training, get you grabbing the rim like, okay, everyone, everyone can put the ball in the hoop. And now like, why don’t we just do a little slam dunk competition? What kind of style you got? What are some tricks, some moves and cool stuff, but that’s all cool. But it, it cannot happen unless students are actually prepared at that level.

Jason Roberts (06:37) What are we doing?

Justin Skycak (07:02) Right? It’s like the problem with projects that we always talk about is like projects are great if you have the prerequisite skills in place, which almost never happens in classrooms. Even in classrooms, supposedly cover these, these prerequisite skills. It’s like, like we just, okay, teacher set it to the class. And I would just check a box that like, okay, now everybody knows this like, Nope, Nope. Nobody like maybe like a 10 % of the class knows it like the

Jason Roberts (07:27) That’s done.

Justin Skycak (07:32) the couple like smart kids and they actually learned it outside of class like last year because they were just reading up on math for fun like yeah.

Jason Roberts (07:39) Yeah, so you get, you got like two, two errors. So there’s error one, which I think was where, uh, most math classes were in historically, and you’re talking, you know, back in the seventies or eighties or whatever. I was just like, okay, the teacher would generally just talk, stand up at the board, talk, and then sign here that do these odd problems in your algebra or your precalculus textbook and go home and you would do them. Maybe you got your phone working with a buddy that guy used to do and

Mitchell, what’d get for 17? What the hell? Oh, man, that’s right. It’s cotangent, not cosecant, or whatever, and you’re working bets. So you’re just sitting there bored, right? Or when they’re younger, you’re giving them worksheets in fourth and fifth grade, and they’re just kind of grinding through worksheets. It’s just kind of boring. And I was good at math, but I didn’t love math when I was in middle school. It was just homework, right? was just like, nobody was like, oh, I love math.

Justin Skycak (08:33) Yeah, that was my experience too. Just basically just doing

problems. Yeah.

Jason Roberts (08:38) Those long divisions, that long division is really something, isn’t it? You know, mean, what? So, but then they over-corrected it for two reasons. I mean, one was like, well, you know, you can go back to the math wars and the common core stuff, and this is kind of happens all the time, but it was like, these students can do these algebra problems or things, really calculus problems, and they can’t apply them to anything. They get your physics class in college and they have no idea what an interval really represents or they can’t really apply, they can’t read it.

essentially a word problem, physics being one of the number one place where you can apply the math. And then they’re like, this is terrible. These students with those math, they can’t use it. So we need to do lots and lots of word problems, which can get really painful when you do a lot of word problems, right? And also become fairly inefficient. So you kind of just spread it out. Or was like, we need to make it more engaging. We need to make math fun. Math is fun, right? You gotta love it. And so then you make everything a project and a class discussion.

but then there’s not a whole lot of skill building going on. And so the way, the example of that would be like, if I was coaching a basketball team, the first instance, all we would do is drills the whole time. We would do shooting drills and passing drills and rebounding drills. And we just drill and drill. We never get to play basketball. And the kids are like, you get home from basketball practice, mom’s like, so how was practice? You’re like, yeah, it was boring. know, it’s like, well, this is a basketball, it’s gonna be fun. You’re like, it’s hard.

You know, like I get my shots a little better, but you you might kind of like it, you know, but then they can’t really put together, but also they can’t really put it together in terms of an actual game of basketball. Right. Basketball is not just shooting free throws or just shooting jump shots or just doing, you know, bounce pass drills or whatever. mean, it’s like, you got to put us all together in the context of game, which is what we’re trying to do. The second is you want to basketball practice and the coach says, all right, play. You guys are like, what? And they just run around like mania.

Justin Skycak (10:04) you

Jason Roberts (10:33) Like, you ever seen a really, really poorly run middle school gym class? That’s what it looks like sometimes. The gym coach would be like, ah, you know, like we had one of these like the baseball, high school baseball coach and they just, and he had a kid on his phone and the kids were just running around. It was insane. Nobody’s learning how to play basketball and it was terrible. And so you get a situation like that. And then the team, they go play a game after first month of practice and they get blown up by 50, 60 points and they, the parents can see that the kids can’t play at all.

And parents are like, what is happening? We just got killed. My daughter cried on the hallway home from the game because it was so embarrassing. You know what I mean? It’s a disaster. So it’s like both of those are terrible. So what you want to do is combine the skill building first. And on that subject building, you build control, you know, kind of, you know, like a controlled scrimmage. And then you open up to open scrimmage, which you do the same thing with projects and things like that. But in a normal class, like you said, you allow the teacher to be human. You allow the teacher.

to do the things that you normally don’t have time to do, more interesting discussions, challenge problems, projects, whatever, but it’s built on a foundation of actual earned skills that they can apply that isn’t like, oh, we got this project, and it’s like one kid in the group of eight who knows how to do it, and the other one, they’re just kind of like, what is that? Is it square root of two? I don’t know what’s going on. And then one kid’s like, okay, I’m just kind of doing this, and everyone else is drafting. They call it freeloading.

which you would see happen all the time, and it doesn’t work. But if you have with the math academy, you can say, I know everybody knows this, because everybody’s got this checked on their stuff, so everybody can do this. This project is appropriate for everyone to do. Anyway, that’s my recommendation. And of course, the last thing I’ll say about it is, in a classroom, can adjust to taste, right? You might have a really social group of kids.

and they’re really interested in having discussions about math or what do they do math in NASA? What did you do when you were in school? They might want to talk more. You might have another group that they love the challenge problems. They don’t really like to talk about as much. Just think, know. Or you have some kids who just like, they got a group of kids. They’re all really have a lot of extracurriculars. There’s a lot of pressure on them. They just want to get their math academy done in class so that they don’t have to do homework because they got soccer practice or drama.

Rehearsal for the play and whatever right so there so you know you do things maybe two three days a week you below the shorter side if kids are kind of like I’m interested when I come in like I Fine you know that’s fine So you can adjust and what you do it up which of fourth or fifth graders could be different what you do with 11th to 12th graders But you can adjust but math Academy allows you the flexibility to know that the students are mastering the skills and you the master

and then you can fill in the extra time, use that extra time that Freedom! advises you to do whatever is most productive, most enjoyable for that or that group of kids. So that’s my, that’s my skill on it.

Alex Smith (13:28) Yeah, I was going to follow on from that actually. mean, there’s certain things that the system can do that even great teacher cannot necessarily do in a classroom environment. And one of those things is like a true implementation of mastery learning. just for anyone, if mastery learning is like a new concept, it’s the idea that you’re not allowed to progress onto post-requisite skills, not allowed to progress onto a topic.

unless you’ve mastered all of its prerequisites. So students have to develop proficiency in prerequisite topics before moving on to post-requisites. And this is very, difficult to implement. A true implementation of mastery learning is very, difficult in a classroom environment. So suppose you’ve got, for something like adding fractions, you might wanna make sure that every student in the class, say you’re planning to do a project where you’re,

It’s about adding fractions with unlike denominators. Well, you need to make sure that students can add fractions with like denominators first, make sure they got that skill down, and then move on to the unlike denominators case once they’ve mastered the prerequisites. once you know that every student has not just achieved familiarity, but mastery in that topic and its prerequisites, that’s the point where you can say, right, I know full well that every student has mastered this topic. We’re now ready to do some kind of project.

on that. So it facilitates the teacher being a human, as you say, you know, sort of like doing some fun, challenging game, but at the same time, you know full well that the students have got the fundamental tools in the bag. I mean, it might seem stringent, but it’s what’s necessary in order to achieve mastery in math topics. mean, math is relentlessly hierarchical. mean, topics

perform prerequisites for other topics, which perform prerequisites for even more topics. And you need to make sure that every topic is nailed before you move on to its post-requisites. Otherwise, you’re really just kind of building on a house of cards and it can come crashing down sooner or later if you don’t.

Justin Skycak (15:25) Yeah, that’s a good point. it’s like, you know, having kids do a project in the class can, if kids have their prerequisites in place, it can be very, very fun. But if they don’t, it can be like the worst experience ever. Because you’re just like, you know, they’re sitting there confused at what are they even supposed to be doing? Nobody has any idea what’s going on. They’re just like trying to do what the smart kid does or whatever, copy off of them.

They come home with homework on this project, but they have no idea how to do. They got to try to get their parents involved. Parents are confused. And there’s no like easy scaffolding, no path to like, how do I actually succeed at this thing? Cause they’re, they’re just missing so many prerequisites. that’s regardless, even if the project is something that they find interesting, like if they can’t actually do it, if they have no hope at it, then that’s a terrible experience.

And in fact, if a project is something in like a content area that they find interesting and they are being continually asked to perform skills and manipulations in that area that they are unable to do, what kind of messages does that send to them? That sends that, hey, you know that thing that you were really interested in? Well, you kind of suck at it, so don’t bother. like, it’s a terrible message to send to a kid. how about that? I mean, you just want to put them in a position where they succeed. ⁓

And if they succeed at things that they are particularly interested in, that’s even better. That’s like the best of both worlds. But if kids are not succeeding, then because they’re being set up to fail, then that just sends the wrong message.

Jason Roberts (16:57) That’s what true empowerment is about. It’s about giving kids the skills they need to do the next thing that they need to do. Now, imagine you’re a teacher and even if you have the best intentions, like, you can do it, come on. And the teacher’s looking at you expectantly, thinking you should be to do it. You know you can’t do it because you don’t understand some stuff because you didn’t really understand what you did the last couple of weeks. You kind of faked it. And then you’re kind of hiding. You’re ashamed. You’re anxious. You’re

You have some real self doubt. have some, it’s a lot of negativity. And that’s why when I, I’ve talked to a lot of adults who come at, come to math Academy, ⁓ you know, much later in life and are using it to skill up or go back to grad school or whatever it is. And they’re like, you know, Jason, I had a really traumatic math education. I’m like, wow, traumatic. I it’s a pretty strong word for math. They’re like, yeah, no, was, it’s, I heard that many times.

I’d never heard math and trauma put together like that. And they were like, yeah, I got to go back to basics. These are people who have degrees from elite institutions or advanced degrees. And they’re like, yeah, I think I really need to rebuild my foundation and go back to basics. And I’m thinking like algebra too. And they’re like, I got to go back to fractions. I’m like, fractions? I’m like, wow. And they’re like, yeah.

Justin Skycak (17:54) Same.

Alex Smith (17:55) Thanks.

Jason Roberts (18:20) And I’m like, well, how did you get through it? And he’s like, I don’t know. I just kind of faked it and kind of, I mean, imagine faking your way through math. I know what happened with that means of faking. It means doing extra credit, retakes, project-based assessment, class participation. But a lot of it’s fake. don’t really mass the material, right? We’re giving these kids opportunities to get a passing grade or at least a good, or a B or something, or.

whatever so the parents aren’t freaking out, meanwhile the kid isn’t learning. It’s like when you read these stories of the kids who graduate high school and can’t read, well how does that happen? That’s a tragedy. That is neglect on top of the educators. But sometimes they’re put in a position where there’s all this pressure, like well you can’t fail these kids, or you can’t, okay well, what are you gonna do? And so anyway, but the kids themselves, the students, they absorb all this

you know, all of this expectation and disappointment when they don’t do well and they just hate it and there’s a lot of self-loathing. It’s just really bad. why a lot of people are really like mad. Whereas like if you just went step by step, it’s like, it’s not that hard. It’s when you miss steps and then you’re expected to do stuff. It’s like, if you get, even if you’re getting B, B pluses and your tests, you know, in fifth, sixth, seventh, eighth grade, of high school, you’re accumulating deficits.

You did not master that material. There are things that you didn’t master. Now there might be some that get picked up on it. know, stuff goes slowly enough and there’s a little bit of spiraling going on in middle school. But you go into algebra and you get some Bs. You know, maybe you didn’t really figure out how to complete the square. Maybe you didn’t really figure out how to, you know.

Justin Skycak (19:56) And any calculus

teacher knows this, right? Like how many kids come in and they don’t know how to complete the square. They barely even remember how to factor. They can just do the special cases. They’ve forgotten the quadratic formula. They don’t know their trig. They don’t even know like their unit circle. They’ve seen it before. They remember, yeah, we did a unit circle thing like last year or the year before. I can’t remember what it was. And this stuff, you need to be like really quick on it.

and just have it automatic. Because if you don’t, then it’s every single calculus problem that should take you like two minutes to do, one minute to do. It turns into this almost like this massive research project that takes like an hour. You can turn ⁓ a solve a quadratic equation. If you know how to do that, if you’re solid on these skills, it takes you what, like 15 seconds? If you don’t, then I can easily take you like 10.

minutes to solve. And that’s just one component of a sub skill within solving an integral, solving a calculus problem.

Jason Roberts (21:01) Yeah, you wrote up like sort of an example of this in the Math Academy way where you had two students and it was sort of like at a pre-algebra level or something. It was really basic and have example, student number one doesn’t know how to do one thing or two or just they didn’t know their math facts. They didn’t know the multiplication. And you show how well one student could just look at it go, oh, they mentally do like almost all of it in their head. It was really easy. Another student was just so much extra work.

Justin Skycak (21:08) Yeah.

Yeah. Yeah. Well, you

can see this even happen. we’re talking about algebra and calculus right now, but this happens even at an even lower level. Like just think about like just an exponent. Say you want to compute four to the third power, four times four times four. Well, if a kid knows their multiplication facts, it’s like, okay, well, four times four is 16, right? And then 16 times four as well. 10 times four is 40, six times four is 24, them together, 64. No big deal.

If a kid doesn’t know their multiplication facts and they’re like, okay, four times four, how much is that? Well, four plus four plus four. Oh, I need to start adding like 12. Yeah. Yeah. So you do exactly. It takes forever. Yeah. Yes. They start finger counting. And now you’re like just so many layers deep into like, you don’t even know what you’re doing anymore.

Jason Roberts (22:05) your finger pounding.

Alex Smith (22:06) Yeah.

Jason Roberts (22:08) unbox this and whatever weird stuff they’re doing because they didn’t really learn how to do the method.

Justin Skycak (22:20) You started computing four times four, but then you realized that, okay, four plus four plus four. And then you’re counting it, you’ve ran out of fingers. You try to remember, somehow you get to like four plus four plus four plus four. You miscount, you say that’s 12 or 13 or something. And then you carry that forward into the rest of the computation and 10 minutes pass. And by the time you pick your head up, you’re like, okay, four to the third power.

57, right? And the teacher’s like, no, that’s not right. Go check your work. And then they’re like, wait, what do you mean check my work? I ran out of fingers. I have all these scratch work on my paper. What do you mean check my work? And then meanwhile, Johnny’s sitting next to you who knows his multiplication facts just really well. It’s like, what do mean, dude? It’s like four times four times four, just 16 times four, 64. What’s the big problem?

And now, like the finger counting kid just feels like a complete idiot, right? So not only do they have to do all this work, they messed up, the teachers are shaking their head. They’re like, what do you mean, check my work? I have so much work. Meanwhile, the kid next to them is like, oh, it’s super easy. now not only are they like…

there’s this skill gap, they’re building up all these emotions around it, like, I’m not good at this. I hate this. This is traumatic. Everybody says this should be easy. This is hard for me. Well, somehow it’s not hard for the kids sitting right next to me. So they just must be a genius and I must be an idiot. I guess that’s what it is. And that’s what they take away from this experience. But you can, if you just have them be solid on their basic skills, like it all goes smoothly, right? You don’t.

I get the sense that they’re dumb when they’re just missing some prerequisites.

Jason Roberts (24:05) Yeah.

Alex Smith (24:06) There was, um, it’s also do with context switching, like whenever you’re sort of instructing a student or, you, want them to remain focused on the task in hand. Right. So in that particular example, it’s four to the third power. Okay. What does four to the third power? Oh, it means four times four times four. So that’s the thing you want them to get. Um, but as soon as you, as soon as they start struggling with their math facts, the content in switches to, okay, how do I do these multiplications? Oh, that’s just repeated addition. So you’ve kind of gone.

in the context of the knowledge graph, you can’t go on several steps back, just to actually, you switch context completely almost from computing exponents down to elementary math facts. And actually, it’s a slight tangent, so I’m working, ⁓ so my son Dominic has started Math Academy recently. And I noticed that in one of the lessons, again, it’s just in regards to context switching. In one of lessons,

Jason Roberts (25:02) You’re like, designed

this crappy lesson? wait,

Alex Smith (25:04) There

Justin Skycak (25:06) Dog fooding.

Alex Smith (25:07) was a moment like that. was like you always say that you got to eat your own dog food. And at this time, I actually ate my own dog food a bit and there was a little bit that tasted a bit like dog food.

Justin Skycak (25:18) Like, man, I don’t

want to take a second bite of this. Like, let’s, yeah.

Alex Smith (25:21) Exactly. Yeah, it was it

was it was like a fractions thing. And it was like, you know, representations of like improper fractions, something like that, or mixed numbers. it was like, the lesson was going well, but the very last knowledge point went from fraction models to number lines. And it just kind of, he was doing really well. But the switching context just completely threw him. And I was like, I was like, oh, this was this was a mistake. It’s kind of I mean, don’t get me wrong. Many kids could

switch context like that, but it’s just like some kids can’t and that presents like a tripping, that presented like a tripping block for him. So yes, I’m planning to go back and split that lesson into a new lesson and do some things. yeah, just to go back to the point though, is that you really want the kids to have those, gets mastery learning again, you want them to have those fundamentals now so they can just focus on the context that’s important right now.

not something which is, you know, you should have mastered and is completely almost completely separate from the thing you’re meant to be doing at that moment.

Jason Roberts (26:21) But in fairness though, Dominic is like two years younger than he’s not in an American fourth grader, right? He’s doing a fourth grade career, but he’s like in second grade or something.

Alex Smith (26:30) I mean, we’re pushing the boat out with Dominic. So he’s in the he’ll be in the equivalent of grade two in in the USA. So yeah, so he’s doing a couple of years ahead and and he has some kind of like content issues with concentration and stuff like that.

Jason Roberts (26:44) Well, who is a man

that’s a second grader, doesn’t have problems with concentration? I mean, I don’t know. have problems with concentration, but a second grader. Yeah. So you mean, you know, but, which actually is good because he’s, because he’s younger and he has, you know, struggles with attention. That means that the, that for the lessons to work for him, they have to even be that much better. Right? Cause it’ll work for him. They’ll definitely work for for the rest of a normal fourth grader at the right age at the, you know, to do that.

Alex Smith (27:14) Yeah, absolutely. mean, this is, this is a, this is a wonderful experience. I actually test drive cause he’s starting right at the beginning, fourth grade and he’s literally, I mean, I think he had on the, on the diagnostic test, I think it was a couple of topics he managed to kind of get placed out of, but the rest of it. So we’re pretty much starting from the beginning and if things go well, I’m hoping to, he traverses the entire knowledge graph. So I get to see him do it. That’d be, that’d be fantastic. ⁓ but yeah, I mean, like I say, for a fourth grader, that actual switching context would probably be fine for many of them.

But it wasn’t fine for him. And so, okay, well, there’s gonna be some fourth graders who that’s also not gonna be fine for. Most of them probably could handle it. Some of them won’t be able to. That’s a problem. We need to kind of split this out into a separate topic. know, so that’s, yeah, high on my to-do list. And I imagine there’ll be a few things, a few situations like that along the way.

Jason Roberts (28:02) Yeah, I think it’s a great experiment for you. You’re making it for all these kids. they’re like, why are these guys and kids struggling with this lesson? And you’re like, oh. But just so people know, we have a lot of tools that we use to analyze student performance on particular lessons and stuff. so not only is this lesson, there’s a pass rate that we don’t consider to be acceptable.

Alex Smith (28:05) Mm.

Jason Roberts (28:28) and then we can analyze where students are struggling and what type of students and then can do act on that. mean, Alex, how many, if you look back, could you even guess like how many lessons that you’ve actually had to either split in two and create, okay, this was one lesson of three or not points out of two lessons of three or four knowledge points that so I develop them independently, give them give each one a little more time or versus say, okay, we just needed to introduce another knowledge point.

We kind of skipped a step. We went step one, two, four, and that without realizing it. first of all, how many lessons do you think across our curriculum, or let’s say after high school, up through grade school, K through 12, or through calculus, would you say that we have been either split or injected new knowledge point? And of those, what was the differentiation, or what was the ratio of split versus augmented?

Alex Smith (29:22) I’ve used a split tool hundreds of times across the curriculum. because we did like a big analysis a few years back where we looked at every single topic on the system and computed the pass rates for them. And we targeted the lowest performing lessons. where the pass rate was beyond, was an unacceptable level, which around now is like the 80 %

Um, pass first time pass rate is at the minimum benchmark we have on now. I mean, the average is something like a 95 % first time pass rate and then sort of almost 99 % part of into attempts is, is, is the average, but it wasn’t always like that. You know, we had some times when there were some really, some, some topics that weren’t in great shape from that point of view. Um, so during that exercise, I must’ve used the split tool hundreds of times because often that’s what it is, is that there’s too much going on in this topic.

too much cognitive load, too much context switching, how do we deal with it? Well, you split. You split the topic into two. Of course, but then that might give you, you might go from a full knowledge point topic into two, two knowledge point topics. say, ⁓ we’ve actually got a bit of wiggle room here now. That actual knowledge point, you know what? I think that could be kind of split into two. That introductory knowledge point that’s got sort two different types of questions in it. How about we split that into, split that knowledge point into two. So create some scaffolding. So,

In terms of ratios, I don’t know, maybe it’s about 50-50, I’m not too sure. But there’s a lot of splitting of knowledge points and there’s lot of introducing new knowledge points and a lot of splitting of topics that has happened over the years to help just to help transit, help students get through it, not cause cognitive overload. Yeah.

Justin Skycak (31:03) Yeah, so you got a much better sense of that now too, right? Like it’s just kind of in your bones or you’re building a topic and maybe your spidey senses start tingling like, I think we need to split this preemptively. Cause I’ve been in this situation like 50 times before and you just have that emotional response of like, if you don’t make it smooth now, then it’s gonna come back in support issues.

Alex Smith (31:15) Yes.

Yeah.

Justin Skycak (31:30) Like you’re to have to like do this change on the, on a live knowledge graph, make it into like a topic switcheroo that, you know, lock the task and it just gets a little more and more complicated downstream.

Alex Smith (31:45) Yeah, that’s right. You have to spot the fact that there’s an issue with the topic or put up with a complaint and then you’ve got to kind of go through some, because it’s a living breathing system. You can’t just change topics just kind of willy nilly. You need to, as you say, lock the topic so that students can’t, if any students have any tasks, those are frozen until you’ve fixed the issue. Then you’ve got to decide what to do, which often involves splitting topics or splitting knowledge points, introducing new questions, rewriting things.

Justin Skycak (31:50) Mm-hmm.

Alex Smith (32:13) So it’s not like a two minute thing necessarily. And you don’t really want to have to that too often. You want to get it right, as close to right first time as possible. And after being for that, after fixing what was probably like 100, 150 topics that had issues, it’s like, after that, but you’re kind of in, it’s kind of in your bones at that point, right? Okay, yeah.

Josh, I reckon that topic there, I think that needs to be split into two. You need to split that knowledge point out using, introduce more scaffolding here. We need a scaffolding prerequisites. ⁓ yeah, so those kinds of things. So that’s on a high level. That’s what I’m doing quite often. I’ve even used, started using that tactic on my team. Now it’s like, you if someone complains about it, it’s like you, you’re making the decision about this topic. If someone complains, you’re the one that’s going to have to deal with it. You know? So I think that’s kind of like helping to kind of spread the message among the content development team as well.

Jason Roberts (33:02) Well, it’s good that you kind of have the, you give them the incentives too. You’re not just the one who has to deal with complaints and figure out what to do. My favorite saying is, you show me the incentives, I’ll show you the outcome. And so it’s like, if the incentive is like, you’re gonna have pain if this thing, if this topic is not scaffolded right, so they’re gonna think a little harder. They’re gonna be like thinking ahead of time, oh, is this gonna be okay? You know, I probably need to.

spread this out or split this topic or whatever. Just real quick, while we’re on this subject, you know, because we just released the differential equations course like a week or two weeks ago. And are there any topics that just come to mind? If none do, that’s fine. But do you have any topics on your mind where that process of like you fear that you had to split it preemptively that you could even describe it would be interesting?

Alex Smith (33:53) not splitting, I kind of did most of that preemptively, but there was one issue that came up.

Jason Roberts (33:58) You even,

even before designing them, you even knew how to lay out the topics. So you’re getting so good, you have like a sixth sense, like, okay, these are the topics and you can separate those out even before even really designing them where they don’t even need to be split.

Alex Smith (34:12) Yeah, that’s true. Or what you find is you start designing, and this is long before it goes live into the system, you start developing a topic. It just jumps out you. This needs to be split. This needs to be an extra. You don’t want a student doing all of this on a single day. You want to split this into two topics. Hang on a second. This requires like arg max or arg min. Do we have a prerequisite topic for that? No, we need to create a prerequisite topic for that. So that kind of all comes out in the kind of the early stages of the course development process.

But just on the differential equations course, was no split, but there was actually a complaint. was something I actually intervened in, but it was just one instance that I didn’t manage to do it. And that was where we have some topics on numerical integration or numerical solutions of differential equations. And some of the computations are quite involved. I thought, multiple choice is probably, when you’re doing like a couple of steps of RK4 or

ABM to or something like that, you

Jason Roberts (35:11) RK4 is a grunge cutout.

Alex Smith (35:13) That’s it, the Runge-Kutta methods and ABM2 is Adams-Bashforth Moulton methods. So basically you’re finding a numerical solution of a differential equation. So it’s kind of emulating what a computer would do, but just like you’re doing like one iteration rather than thousands, which is what an actual computer would do. And I was thinking, well, you really want multiple choice for these kinds of questions because there’s lots of numerics involved.

You know, they might have a perfect understanding of what’s going on, but they might forget to carry a one somewhere and all of a sudden, or, you know, all of a sudden they get to the incorrect answer. And it’s a bit unfair because they do actually know what to do. They just made a slip up on the arithmetic. And, you know, you don’t really want to get to the point where you’re inputting a free response answer to realize you’ve messed up. It’s like, you want to get through to the answer.

compare it against the multiple choice and go, actually, hmm, something’s not right here because I can’t see my art here. I see what I did. I went back and I messed that up. Now there was one instance where I missed the multiple choice. It was still free response. And it was one of my colleagues’ decision.

Sure enough, he did. came up on the diagnostic. He got offered a free response version of the RK4 and he posted about it on Twitter saying, I guess it builds character, know, of after seven steps, missing it, forgetting to carry one. We’ve been told you’re wrong. Okay, really sorry. I’ll go back and fix that. And we did. was, it was a quick fix, thankfully. but that was, that was, but I’ve said I preempted, but just, yeah, because we have quite a few of these types of questions didn’t matter. Didn’t catch that particular, but I said to Josh, I said, Josh,

Justin Skycak (36:26) You

Alex Smith (36:44) Next time that happens, you’re gonna be dealing with it. So, you know

Jason Roberts (36:49) Well, you know, it’s the best thing about, you know, like Twitter or X or whatever these social media is that they hold you to account, right? Keeps you honest because if something sucks, they’re gonna let you know. I mean, some people will be kind of subtle about it. They’ll send an email and might say, hey, you know, I think this might have been a little unfair. This is new. Like some people are gonna put up on Twitter. mean, that guy had, you know, it builds character. I mean, he put it in the good humor, but, you know, we live in a world now. They’re going to show you.

your ass, right? They’re gonna be like, you suck, this sucks. And so you don’t just get just like, oh, here’s the textbook. And if it’s crappy textbook, and there’s a chapter eight was kind of lame, because you just didn’t have the time to go through it. And it’s like, well, whatever no one’s notices and kids suffer for 10 years, until there’s a revision. We don’t we find out immediately people, people complain.

Alex Smith (37:18) you

It’s a hyper accountability. There’s no hiding at all. When we first started becoming, when we first launched and started getting quite popular on Twitter, I was kind of shocked actually. People are posting questions and lessons snippets on Twitter. was like, my God. And there’s another thing as well actually these days, they can just literally take a copy of your thing.

Justin Skycak (37:56) You

Alex Smith (38:06) question and post it into an LLM and ask its opinion. It’s like, well, the LLM says you’re wrong or there’s not enough scaffolding here or whatever kind of thing. So there’s this whole new layer of accountability, which is coming. We’re now accountable to them. It’s available. They can post anything we create can be posted anywhere. They can now ask LLMs for their professional, for their input. So yeah, there’s no hiding at all. So you really have to get this stuff right. It’s like first time, you know.

I mean, a little bit of fallout like that, easy to fix, you can manage, but you certainly don’t want that happening too often.

Jason Roberts (38:39) Yeah, that’s really funny. Jess, before we get back on the education stuff, I’d just like to ask Alex one other question, because we’re talking about it. So the differential equations course came out. I’d like to just tell us a little bit about the course, like how you constructed it and what you were thinking and how people should think about the course. mean, what’s it comparable to? I mean, I think you mentioned you thought it was one of our best courses your team has ever created. Is that you agree that that’s true?

Alex Smith (39:07) I mean, I kind of come from like a mathematical physics background. So I am sort of somewhat biased. I’ve wanted this course for a long time. So I was super excited when it when we finally managed to launch it. So I suppose the problem with a lot of differential equations courses from my perspective, it can kind of be seen as like a hodgepodge of different unrelated techniques. OK, for this type of differential equation, you got to use this trick. And for this, you got to use that.

And it’s just kind of a, it’s very hard to kind of, for some people it’s hard to kind of gel it all together. But I’m not talking differential equations for many years. And I’ve always felt that it really is like a beautiful coherent subject if it’s like approached in the right way. So just to give you an example, so at this very, very elementary level, lot of the techniques are

A lot of the things you study are in relation to linear differential equations and first order linear. If you look in many textbooks or look in many courses, first order linear seems to be like this whole class by itself because there’s a particular method which is available called the integrating factor method, which you can use to solve any first order differential equation. Now, it doesn’t generalize particularly well to higher order equations, but what happens is that

People tend to sort of go straight into the integrating factors and they never really explore properties of first-order equations that can be generalized to to higher order equations like things like There’s really great opportunities for scaffolding. So things like, you know the superposition principle and you know, they the Sometimes people put the structure theorem like, you know

The general solution is the complementary function plus the particular integral. mean, this often gets introduced later on down the line, sort of second order equations, but it’s completely applicable to first order equations. And that’s where it should be taught because that is where the setup is the simplest. So you master that kind of thing for first order equations and then applying that to higher order stuff is pretty, it gives it more context.

What else would I say?

Jason Roberts (41:13) Well, let show you this. So when we’re creating this course, know, thing I would kind of say to you, back in the group, back when we first created, I think our linear algebra course, which might’ve been our first sort of advanced post calculus level course. And the sort of the thing, the target that I was shooting at that I wanted you guys to hit was this would be a course that you should be, you would see at an MIT, you know, Stanford, Harvard, some elite.

Program so you want someone I don’t want anyone to come and look at our course and be like well At Berkeley we do it we do it for real like this is you know kind of third-rate Junior college stuff or something. I don’t know. It’s like I I was like the hell no, that’s not happening. Like I want it To be like you look at and go damn like wow, they’re covering all that stuff, right? Cuz I don’t I I can deal with people saying your stuff’s too hard. I can’t deal with saying your stuff’s lightweight. It’s income

Justin Skycak (42:07) Comprehensive.

Jason Roberts (42:14) It’s it’s low grade. I’m not going to do that. That’s not going to be our error. I remember initially when you did that, I said, you just take a look at what would be considered an elite treatment of this course of a concrete version of linear algebra. It wasn’t a proof oriented, so we’re going to come up with a proof oriented course later, but just a concrete treatment of it, a computational oriented treatment.

And you said, and so you were looking at all of the most popular textbooks, university-level textbooks on linear algebra. And then I remember you had like a spreadsheet of like 10 or 15 universities in their syllabus. they kind of looking at their, because a lot of them have, so the syllabus of public or their final exams from like two or three or four years of being public. And you can kind of get a sense of what major topics I covered. And you kind of had a whole spreadsheet of like, okay.

Now, did you do something similar to that for different equations? mean, I you’ve taught this course yourself at the university level, like said, in many years, so you already had a pretty strong sense of what the course should be. What kind of research did you do when constructing?

Alex Smith (43:20) So yeah, so one of the primary things you do look at is what the elite level universities are teaching. And so yeah, so we create like a big spreadsheet of five of the top 10 universities in the US plus a few from the UK and a few other places. And just try to create like a superset of those things. What would you see in an elementary differential equations course, which is a kind of a superset.

more or less of all of these elite, you’d find that these elite institutions. So that’s what we tried to do. And I also brought into some of my own experience into it. So we push a little bit further on some of the things you might not necessarily see in like an introductory differential equations course, just to give students a little bit of a flavor of what’s next, you know. So for example, we went in a little bit into the numerical analysis stuff. instead of doing like Runge-Kutta 4 or RK4,

We also did some numerical analysis, some stability analysis, things like that. So if you want to go down the numerical analysis route, which is a whole other thing, you’ve got some kind of foundation. ⁓ So how quickly do these things converge? Do they always converge? Do they blow up? That kind of thing. How efficient are they?

And also something that isn’t discussed in many elite institutions in an elementary course are things like boundary layer problems. and this is really the foundation of partial differential equations, which would be, okay, probably one or two post-requisite courses later, but here, the theory of, ⁓ boundary layer problems are.

much richer usually than what you would typically find in regular differential equations course, is typically initial value problems. The solution structures are much richer. They’re used as a basis to construct much more complicated solutions, much more complicated problems in PDEs, in physics and things like that. So it was finding that superset, but also kind of nudging things in a couple of different directions, what I think we really build on in the future.

Justin Skycak (45:18) Yeah, differential, because differential equations is one of those like, it’s one of the, one of those highly branching subjects within mathematics, right? That’s like, can, there’s, what do do after it? Well, there’s like so many different directions you can go into because it touches on, I mean, basically everything in the world that can be modeled mathematically, like involves differential equations to some extent. Right.

at least that can be modeled with continuous mathematics will involve differential equations to some extent. But then there’s also like pure mathematics concerns.

Jason Roberts (45:52) Well, Feynman,

there’s this thing that Stephen Wolfram wrote up about how Feynman, he and Stephen Wolfram and Feynman, Richard Feynman, Richard Feynman was one of greatest physicists in the modern era, went and they spent a summer at thinking machines. And Wolfram has this, like a copy of the math that Feynman was doing to model what the thinking machines was doing.

discrete machine and he was using differential equations and calculus to model, which was, Steven Fulford was laughing about. He’s doing these old school calculus. It’s like you can even model a discrete system using differential equations. If you’re good enough, guess, with the differential equations, they make a few assumptions. That was so funny.

Justin Skycak (46:25) You

Yeah, that’s what physicists will do all the time, right? That’s like the favorite tool is whip out differential equations and everything, anything. Yeah, because they work so well, because they’re so powerful. Yeah, so it’s really, really valuable to just see all of the different branches, see a little down the path in this kind of introductory course.

Alex Smith (46:40) Okay.

Jason Roberts (46:57) Well, you posted on X, you showed this, the Thanos, know, gauntlet, right? And it had like the five stones. was it?

Justin Skycak (47:02) Yeah

Yeah,

yeah, let me, let me pull it up. Yeah, right. Cause this was this differential equations course was like the last course in our core undergrad math sequence. I mean, we’ve got, we’ve got basically everything.

Jason Roberts (47:23) So you have multivariable calculus, linear algebra, multivariable calculus, probability statistics, discrete math, differential equations, And the methods of proof. But methods proof, that’s sort of more towards a pure math.

Justin Skycak (47:32) That’s right, yeah.

Yeah, it’s more for sorts of pure math, right? So it’s like, right, the infinity stones, basically single variable calculus, multivariable calculus, linear algebra, probability and statistics, and differential equations. I guess there’s technically a sixth infinity stone in the Avengers, right? ⁓ But ⁓ in the image that I posted, there was just five visible. So it’s like, okay, we’ll do that.

Jason Roberts (47:55) Hmm

That’s cool.

Well, Alex, let me just say, so are you, if you had taught this course, are you happy with the course, with this, like, this is what I wish I could have taught, had I had all the materials of, you know, when you’re going back to when you were instructing at the university level, mean, does it satisfy, would your earlier self be proud of this? Like, hell yeah, that’s what.

Alex Smith (48:18) Yeah.

Yeah, absolutely. I think it’s one of our best courses to date. always looking ahead, I’m super excited to kind of build on top of what we have as well, like in a second differential equations course or PDE’s course or more mathematical physics stuff or mathematics or physical science, all of it is going to use this stuff as a basis. So there’s so much potential that can come from this course.

Jason Roberts (48:44) Right.

Alex Smith (48:48) But as a course by itself, super happy. I’m super happy we managed to touch on so many different areas and do so in a concrete way. I pick up any book on numerical analysis, and it’s a nightmare to find anything really concrete. It’s very, very abstract. But we’ve done a fantastic job, I think, doing some introductory stability analysis and error.

error, error and stuff like that. if anyone’s interested in that kind of thing, this is, this, this course is a really good place to start to kind of get a leg up on, on, on those kinds of things.

Jason Roberts (49:20) Yeah, that’s really cool.

Justin Skycak (49:20) I think

also just juxtaposing this course against the way that differential equations is typically taught. I I think differential equations is one of the worst taught courses generally because there is this big breadth to it, right? Or there’s all these different branch off points. typically, what I’ve experienced in my own education and in many students that I’ve tutored and that…

people that I know is like, well, typically, instructor will kind of like, speed up to, you know, get through your integrating factors and linear differential equations, then go do characteristic polynomial second order. And then it’s like, okay, we covered the basics, let’s just go into my favorite area of research. And then you go do a bunch of that. And that area might be like, you know, maybe it’ll be predator prey models, oscillations, looking at like, bifurcation.

theory, whatever phase diagrams if they’re more applied, or maybe it’ll, it’s just going to go into like, you know, existence and uniqueness and, and, know, the theoretical concerns about this, or maybe it’ll be numerical stability analysis, but it’s like, just depending on who your instructor is that you can get a completely different view on what, what is even the subject? What does it cover?

Like you can, if you get an instructor with just like very pure concerns about differential equations, you may never realize the extent to which it can model stuff and how to actually approach that. you know, so actually in my own undergrad, differential equations was taught within an abstract algebra course. It was so wacky. It was so weird.

⁓ I think they were doing some kind of like revamp of the math program where they were trying to like make some courses integrated and stuff. And I don’t know if I just like caught that at the wrong time or things were just chaotic. But basically what happened is like that was abstract algebra. The course was called algebra one abstract algebra one. So, ⁓ and, and, and so the way a differential equations was introduced, it was just kind of like, well,

You know, we’re doing abstract algebra. Why don’t we talk about metric spaces? Metric spaces are, I guess, an algebraic object equipped with properties. This is a distance metric. by the way, like, you know, calculus? Well, you can do stuff like that with different equations on metric spaces. Here’s a few theorems about different equations. And here’s, here’s a complementary versus particular solution, blah, blah, blah, blah. Okay. Moving on. And it was like, no, no sort of applications at all. Cause, cause

pure abstract algebra professor doesn’t like these applications. Yeah, exactly. It was terrible. was awful. so fortunately, kind of like, I mean, huge math nerds. I’m a kid who like, reads about, does problems and differential equations outside of class. So I came in with a lot of background information on that. And I had been seeing this kind of in the context of physics and stuff. I was like,

Jason Roberts (52:04) It’s messy world stuff, yeah.

Justin Skycak (52:28) I was fine. didn’t make that big of an impact on me, but basically everybody else in the class who had not covered differential equations on their own was just being hung out to dry in this awful differential equations course. Yeah, it was so bad. ⁓

Alex Smith (52:46) Yeah, when

you said that, I was like, I was reading your description. was like, what? But yeah, but I felt for this particular, I felt it was like, as I’ve touched on already, I felt it was important to get like a flavor of obviously going to go into a lot of detail about the things that you’re going to find. but touch on everything that is of interest, know, sort of systems of differential equations. We’ve got quite a large section on that. You know, the plus transforms Fourier series.

Justin Skycak (52:49) Yeah.

Yeah.

Alex Smith (53:15) boundary value problems, numerics, actually finding numerical solutions, analyzing the numerics, some special cases, know, homogeneous equations and stuff like that. So yeah, to really kind of touch on everything. later on down the line, when we have a course on numerical analysis, we’ve already got a strong foundation.

When we go into PDEs, we’ve got boundary value problems. We’ve got Laplace transform. So that’s already there. So any direction you want to go in, we’ve already got some kind of foundation in this course, which I felt was really important.

Justin Skycak (53:48) Absolutely.

Jason Roberts (53:48) Yeah, that’s

great. The course looks great to me, and I’m really excited we finally have it. This is awesome.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.