Link to Podcast

0:00 - Introduction
4:00 - Applying the MA Way to X Growth
7:40 - Status of the ML Course and its Kick-Ass Coding Projects (Part 1)
25:50 - Jason's Near-Infinite List of Important Things
34:20 - The ML Course Has Been a Massive Undertaking
42:10 - Breadth-First Development
44:30 - Status of the ML Course and its Kick-Ass Coding Projects (Part 2)
50:15 - Why Math Academy Needs To Do a CS Course
56:45 - The Never-Ending Stream of Confusion
1:00:30 - The Story of Eurisko, the Most Advanced Math/CS Track in the USA
1:24:20 - Intuition Through Repetition: Machine Learning Edition
1:29:40 - The Importance of Spaced Review
1:43:30 - Upcoming Course Roadmap
1:47:40 - Spaced Repetition 2.0: Accounting For and Discouraging Reference Reliance
1:54:45 - Overhelping: A Pathology of the Over-Involved Parent/Tutor
1:59:21 - Yes, You Need to be Automatic on Math Facts (and Yes, Rapid-Fire Training is Coming)
2:04:55 - What Happens When Students Don't Know Their Math Facts
2:05:50 - The Horror of Attempting to Teach a Class When Students Have Multi-Year Deficits in Fundamental Skills
2:11:30 - Integrating Coding Into the Math Curriculum
2:18:00 - Combining Math and Coding is the Closest Thing to a Real-Life Superpower
2:18:55 - Creating a Full Math Degree and Getting Full College Credit
2:22:15 - The Power of Pre-Learning: The Greatest Educational Life Hack

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

Link to Podcast

The raw transcript is provided below. Please understand that there may be typos.

Jason (00:00) So this is our first attempt at a Math Academy podcast. ⁓ We’re going to answer some questions ⁓ that Justin. ⁓

Received on X, right? So you. ⁓ You posted something yesterday and you said hey, we’re going to do a Q &A and got how many questions you get.

Justin (00:22) Last I checked it was like around 60 or something. Wow. Yeah, there was a lot. I was kind of surprised by how many. 61. 61 questions.

Jason (00:33) Okay. Well, I don’t think we’re going to get through even a fracture to those, we’ll do our best. I think we should try and do is just address the top ones that you were kind of talking about earlier. You mentioned that there was some frequently asked questions and we’ll get through what we can. And then, you know, of course we can record a couple more shows. I guess we call this a show or an episode or whatever. And

We’ll just see what happens. But I think it’s probably good if we can answer a lot of these. I’m sure that most of these questions are becoming commonly things that a lot of people are confused about. So yeah, let’s just let’s just do it. So what do we what do we got? Oh, by the way, so one thing I want to say is in addition to the Q &A. So if this works, if people like these and they find it helpful, you know, we’ll we’ll not only post, we’ll not only like.

Posted stuff on X and we’ll kind of maybe slice it up into like, know, kind of themed segments or something, or even specific, you know, in-depth answers to specific questions. But, you know, we were talking a little bit yesterday about bringing Alex on, our director of content, maybe, you know, we can do a variety of things. We talk about kind of what we’re building and what we’re thinking about and just the trials and tribulations of

creating a startup and all that stuff, just depending on what people are interested in. Anyway, so I don’t know, any thoughts before we get started on this?

Justin (02:14) ⁓ I’m excited to see what this turns into. This is how things always start, right? With one push, get some positive reception, still figuring out exactly what it is, and we’ll just see how goes. But at the very least, we’ve got some good questions to answer today. ⁓ And ⁓ a lot of stuff that we’ve been thinking about, too. It’s funny, there’s quite a few questions that’s like…

I’m like, wait, has somebody been listening on on our phone calls like the past few months? Cause this is the kind of stuff that we’ve been talking about. And, ⁓ but of course, like none, don’t, nobody knows about this. So it’s, it’s nice to actually just, ⁓ have, a con make some of this conversation that we have, just make it more, more, ⁓ public. everyone, everyone knows what’s, what’s on our mind for the future of the product.

Jason (03:06) a lot

of interesting stuff. think it’s interesting stuff. ⁓ I would be fascinated just from like a tech startup perspective, but I met also, guess, as a student or customer, would probably things I want to know about. And it’s always kind of cool to look behind the scenes, the workshop, walk into the workshop and see the stuff that’s on the cutting room floor and, you know, and be like, what is what is going on and stuff? So, you know, I’ve been encouraging you and Alex to share as much as

of the stuff you guys are working on is possible. Cause every time we have conversations and I’m like, that’s cool. That’s cool. That’s interesting. You should post that. And they’re like, really? Yeah, post that. That’s cool stuff. know, and we’ll, and I, lot of stuff that we talk about, I’m like, man, this stuff is, I don’t know. I mean, maybe I’m just interested in it I’m interested in it and nobody else cares, but we’ll, I guess we’ll find out. But I think there’s an opportunity to share a lot of, lot of interesting things. So.

Justin (03:59) Well, seems yeah, but posting what we’re working on definitely, definitely worked. Cause I, remember last year when I was like at 27 Twitter followers, right? And you were like, Justin, 19.

Jason (04:12) This was in like the end of June or July of 2024. So a little more than a year ago. And I was sitting up in the Sheraton and Palo Alto because my youngest was at the gymnastics camp at Stanford and I’m just sitting there lobby working. And we had this conversation and we’re like, well, we got, we should, we should post on X or something or Twitter. I think it was Twitter at the time, but you know, we need to do something and then listen to the wall.

Dude and individual accounts and and then I looked at your account. had 19 like seriously. 19 I think it’s even possible. mean, do you like automatically start with like 30 or something? But now you have what like 27,000? I mean, it’s crazy.

Justin (05:01) Yeah, yeah, something like that. Yeah, yeah, it’s been insane. always thought I was gonna be like a back office nerd for just the foreseeable future. yeah, I don’t really like

Jason (05:13) You’re

a front office nerd, now you’re a back in front office nerd. You’re an all office nerd, right?

Justin (05:18) Yeah, that’s right. Nerd on all fronts. Full nerd. That’s right. Full nerd.

Jason (05:23) Full

spectrum nerd. No, but yeah, you’ve crushed it on X. I mean, we had a little bit of a, I had a little bit of a competition there to start. I said, let’s just kind of just, just for fun, let’s just see who can get to, I don’t even remember what it was, a thousand followers or a couple of, I don’t even know. I think I had like a 400 or something, 435 if I recall. I think our math academy count itself that Sandy has been running had like 200, like barely, 201.

Alex had, I don’t know, a couple hundred or something, you had 19. And then you just taught us all a lesson. You were like,

Justin (06:02) Yeah. Well, consistently posting, Consistency, get reps, lots of reps early, refine everything we talk about. I’m just applying the Math Academy way to Twitter. That’s all I’m doing. It’s just the recipe for everything. So.

Jason (06:03) If they count on you, step-

I was thinking that this morning while I was getting ready and I was thinking, I thought they’d see the exact thing. It’s like The secret to success in life is consistent effort. You don’t have to, whether it’s exercise or learning math or learning a language or whatever the heck it is you’re trying to do. It’s like you don’t have to do the superhuman effort thing. Just get started and then make a consistent push every day, even if it’s only 15, 20 minutes.

or even less in your case, I don’t even think you put that much time into X and it’s just been like this, you know, so yeah, yours is just a definitely a perfect illustration of the principle. So yeah, it’s amazing. a, you blew us all out of the water. I mean, it was a tight race a little bit at the beginning and then you and Alex kind of just took off and then Alex kind of just got distracted and you just.

Justin (07:18) Yeah.

Jason (07:18) Anyway, enough about this. Let’s let’s answer some questions here. We’re like, Jason, like, what? cares? Like, we answered my damn questions. OK, let’s answer those questions. what’s that? What we got up first? And by the way, I haven’t seen any of this stuff. I told Justin, like, don’t show me anything because I can’t read and talk at the same time. And I know you’re distracted. And don’t I don’t know. We’ll just just go. So what do you got? That’s number one.

Justin (07:40) Yep, yep. All right, so a large chunk of the questions kind of fell into the general category of ⁓ when is this thing coming? People are just, they want to know when is the machine learning course coming? When is computer science coming? When is ⁓ purchasing price parity and student discounts coming? When is streaks and animations and all that? So.

I mean, this is all stuff that we’ve been working on is that various stages of completion. Some are like almost there. Others are like halfway there. It’s kind of like we got a bunch of progress bars loading all at the same time. They’re not happening in serial. They’re all kind of in parallel. yeah, I we need to.

Jason (08:26) So those are good questions. I want to know when they’re be ready to when is this stuff gonna be right? That’s a good question. Yeah. Okay, so which one do want to start first because everyone answered everyone is different, right?

Justin (08:38) Yeah,

why don’t we start with the machine learning one? That’s the one that most people are wondering about. And that also happens to be the one that our progress bar is pretty far along on.

Jason (08:49) Yeah, okay. So, I guess we can answer directly and then maybe give a little background on the course. So, what do we have left to do at this point? From your perspective, from a kind, well, I mean, Alex is a non, but you’re somewhat familiar with what’s going on with the content.

Justin (08:57) Yeah.

⁓ Content is basically done. think they just have like a couple hanging topics, but project is done, content is basically done. We just need to upload it, the UI.

Jason (09:23) Julian is the one in charge of that course now, right? And he’s been posting stuff. get updates in whatever our machine learning channel. And I get, and so they’re sort of just refining some lessons. Is that basically what?

Justin (09:38) Yeah,

yeah, I think something like that. yeah, content is basically done.

Jason (09:43) ⁓ And then there are the projects. How many projects are there?

Justin (09:47) Uhhh, Ballpark 20?

Jason (09:51) Right. that, okay. So, and that has been a massive effort in and of itself.

Justin (09:58) Right? Yeah.

Jason (10:00) A massive effort. You know, OK, so when we talk about the machine learning course. It’s part math. Part coding, right? So it’s not just like our normal math course where we have free response and multiple choice questions. ⁓ You know, and I think when we were first conceptualized in this course, you know, we’re like OK, so if you took this at MIT or Harvard or Stanford or some elite.

institution, how would this course be taught? Because that’s what we’re competing, that’s what we want to be as good or better at. We don’t want people to go, well, yeah, Math Academy’s okay, but at Caltech we do this and this is the real stuff. And like, no, no, When you do our stuff, you’re be like, damn, this is hardcore. This is the same stuff that I saw at Pick Your Elite University.

And what they would do is it would be very mathematical. They wouldn’t just be like, hey, just install PyTorch and do this little go through these steps and woohoo look, you got a handwriting recognition thing. You know, it’s like, no, that’s that’s fun and all, but that’s not what a machine learning course at MIT.

Justin (11:18) Not just import solution, run solution. That’s not really machine learning. That’s like import from a library, software engineering, but we’re like, you actually building up models yourself.

Jason (11:35) That’s right. So you’re building up the math and you’re building up the, you know, you do a lot of supporting math in the math for machine learning course. mean, you you might be one of a very select few who come in and have all the mathematical background to do machine learning. The reality is that very few people will likely come in with that. will, even if they have taken some advanced math courses, unless they took them like.

Three months ago, it’s probably degraded and they don’t remember all of the multivariable calculus on their algebra and probably the statistics that you’ll need to do this course, to handle the material in this course. So it’s no joke. so ⁓ anyway, so we had to do all the mathematical oriented stuff, but then in the theory stuff, but we had to do projects.

Because if you were at an elite institution, you would probably have at least, probably at least three to four big projects, depending on how they did it, how they organized it. They might have a project every two or three weeks. Someone might have one big project that has broken into multiple parts, kind of make them group projects or whatever. But we’re like, okay, we like to do things a little more bite size.

And so we’ve created 20 projects, which maybe got little carried away. I don’t know. Probably.

Justin (13:03) I don’t, you know, I think that’s actually a good number. ⁓ now the thing is like when, when normal people, when you, when you normally think of a project, you think of like a week long, sort of like this takes you, I don’t know, 20 hours to do each project. That’s not what our projects are. Our projects, cause students are going to be coming in with the fundamental knowledge and it’s gonna, the project’s going to be like the, the, the minimum example of a real thing. So minimum example of a.

⁓ neural net classifier. We’re not scaling this up to like ⁓ hundreds of thousands of input data. is just like a

Jason (13:44) We’re

going to to spin up a GPU cluster on AWS just to run.

Justin (13:47) ⁓

Yeah, right. Just like, let’s simplify it into something that exhibits like, ⁓ all the mathematical intricacies gets you building up something, but something that you can build rather quickly. And if you have the right foundations ⁓ and you seek some kind of cool result out of it, some kind of payoff that lets you know, like, okay, this is a real thing that I did. I didn’t just get some like,

82 % accuracy, woohoo. Like, no, no, no, there’s like an actual takeaway payoff. We should probably go through some of these project names, but the idea is, yeah. And now we got these projects spread out over like the entire curriculum. It’s not just one neural net project. It’s not just one neural net and one decision tree project. We’ve got it peppered all the way across the entire, like, if you go up the tree,

Jason (14:25) That’s 100 percent.

Justin (14:45) And it branches until all these branches, like at every leaf, every leaf, every module is kind of like a project that brings it all together. And so each project, ⁓ like this, this may sound surprising to a lot of people, think, but each project should really only take like one class period worth to get through because you’re going to be coming in at the point when you know your foundations, you’re strong in your foundations. You’re not going to be trying to like learn back prop as you’re building.

the neural net. No, no, no. We are covering that prop for it. We’re taking that out of the project context first, getting you strong on that. By the time you come into the project, you’ve got everything ready to put together. You got all the blocks. It’s kind of like we’re not throwing you in the ring with an opponent and expecting you to figure out how to punch and kick on the fly. ⁓

Jason (15:38) or a jab or whatever. It’s like, you already

got all the tools. You got the tools. Let’s just put them together and do something cool, right? Because you’re like, why do I need this? Why do I care about this? ⁓ you want me to show you? Yeah, OK, let’s do something super cool. So yeah, I’m super excited about the projects. are, I think, Gary, why don’t you go through some of the names of them?

Justin (16:05) Yeah, yeah.

Jason (16:07) Because I’ve read them out or given to a few people and I think the people are pretty excited.

Justin (16:17) So all right, all right. So ⁓ these are in no particular order. They’re kind of jumbled up. let’s just go down the list. We’ve got optimizing a drug molecule with single variable gradient descent and momentum. So the idea behind that one is like you’ve got a,

Okay, you know, I actually need to pull up these projects. Cut out.

Jason (16:43) Okay, no, you

know, you have to cut out this fine people. It’s hard to remember everything you’re doing and working on.

Justin (16:52) Yeah. Okay. Yeah. Okay. So the idea behind this one is that you’ve got a ⁓ drug and you’ve got a drug candidate and you need to ⁓ adjust its 3D structure so that it binds to a virus protein, like a key fitting into the lock. And so you’ve got this function that represents ⁓

Like it’s a function of some bonding angle of its 3D structure. And you’ve got to minimize that function ⁓ to find the appropriate angle for this drug molecule to fit into the virus protein and render the virus incapable of all the bad things that it would otherwise do. ⁓

Let’s see what else we’ve got. Stabilizing a search and rescue drone with multivariable gradient descent and momentum. That project is something about figuring out what forces to apply on a drone camera to keep it stable in the face of wind and vibrations and stuff. ⁓

There is another one, ⁓ modeling super ionic ice on Neptune with neural networks. That one is a particularly cool one ⁓ because the idea behind that one is that you’re actually modeling a neural network to learn ⁓ some kind of complicated physics of the situation that would be otherwise very, very hard.

to very computationally expensive to run like these kind of physics simulations over and over again. But if you train a neural network on the data, on the results of those computations, you kind of get it to learn, ⁓ to compress that information down to a space where you can evaluate the neural network ⁓ a lot more efficiently than it would be to just simulate the entire physics of this thing over and over again.

⁓ We’ve constructing an evolutionary tree from ancient DNA samples with hierarchical agglomerative clustering. ⁓

Jason (19:18) I don’t even know what the hell that is. sounds cool, but I have no idea what that is.

Justin (19:21) Yeah, so it’s like, say you’ve got, I think this is the ice stage related one. Yeah, let me pull that up really quick. This one’s a particularly cool one. ⁓

Yeah, yeah. So it’s like you’ve got a bunch of DNA. It’s like, imagine you’re a scientist and you’ve got a bunch of DNA sequences from various ice age organisms and you want to like measure how similar they are genetically. you want to basically construct like a family tree for all these organisms. And so you can do that using hierarchical agglomerative clustering, which looks at the

Well, it operates on a distance metric. So for these DNA sequences, we would be using edit distance. How many, what’s the minimum number of ⁓ letter changes that you need to make to turn one into the other. then, yeah, and then using that information, you can build this evolutionary tree, see which organisms are more closely related, that sort of stuff. ⁓ Yeah, that one’s cool because we got like,

cave lion, cyber tooth tiger, dire wolf, cave hyena, wooly mammoth, and a bunch of plants just to demonstrate that animals and plants are very different. now the DNA- Okay. Yeah. ⁓ okay. was just gonna say, one caveat here is the DNA sequences here are simplified, they’re kind of, they’re fictionalized. real, like if you were to do this in real life, ⁓

Jason (20:48) Oh go on, go on, finish that.

Justin (21:02) you’d have a massive DNA sequence. But the genomes for these things are just so incredibly long. ⁓ Yeah, my wife actually works a lot with just bacterial genomes. Not even like human genomes, not even like, not big organisms, small organisms. And just- Yeah, yeah. And these genomes are just gigantic. Like, yeah. ⁓

Jason (21:23) little wee little organism.

Justin (21:32) She in her lab talk about terabytes of data as though they were megabytes almost. It’s so funny. Yeah, yeah, just throwing her out like, ⁓

Jason (21:39) That’s on terabytes of genome.

To

3 megs PDF or something or 10 megs, no, no, no, this there. 10 terabytes of.

Justin (21:47) Yeah, yeah, exactly.

Yeah, yeah. So we’re not going to subsidize this kind of genetic analysis on our system here. We’re going to just cut down, like, just simplified versions of the DNA sequences. But the point is that, like, the process is very similar.

Jason (22:10) solving the techniques you’re going to use. It’s a toy representative.

Justin (22:13) Yeah, yeah.

problem. Yeah.

Jason (22:20) A toy toy problem is representative of the actual problem, but it’s something that you don’t have to go and do a summer internship at a lab to even be able to approach it. They have the tools and the data and the expertise to even interface with it. Well, OK, let’s let me ask you a quick OK. ⁓ I know we got we got to answer the actual question probably would affect, but before you answer the actual question of when the hell is this thing going to be available is.

Justin (22:36) Yeah.

Jason (22:48) Each project has a number of steps to it, a number of sub-problems. What do they range between on the lower and upper end? Do you have a?

Justin (22:58) I roughly like eight to 13. Kind of like in the range of a normal multi-step, really.

Jason (23:05) Which we’re going to rename as projects as well. Projects is going to be, I think it’s a better name that people understand what the hell you’re talking about because they’re like, well, aren’t most problems multiple steps? So what the hell? ⁓ So each one would you start out and it’s kind of simple on the simpler at the beginning and you write some code or you solve some simple problem and then you build on top of that with more complex problem solving and more right layers to it.

Justin (23:35) Yep, yep, that’s right.

Jason (23:37) Okay, okay, so the real outstanding to do item. Well, okay. First, we can just, it’s mostly me at this point, right? It’s mostly on me because two things need to happen. We have to transfer all of these projects that are currently in like, ⁓ Jupiter notebooks or something like that. What do you guys, what have you guys been putting this together?

Justin (24:02) and Colab notebooks.

Jason (24:04) Is all

you know all the code is Python and. We need to move those over into the system into a project. And there’s some there’s some stuff that I need to do so that that is possible because there’s in. The biggest piece being that support for coding. Right, which I did. I did some proof of concept. I’m more than proof of concept. I got some early versions of.

coding and evaluating and error handling and all this stuff that you have to do to build an in web app. You know Python, you know, editor, know, and evaluate, you know, an evaluation system. And I did that back in. I don’t know when was that. That was back in November or something. And I had I got pulled off it because I had to work on some other stuff.

So I need to hop back on that and kind of get that last 10 % of it done so that we can actually put these police projects into the system and start testing them out. then there’s like the sort of the metadata process that you and Alex and have to go through question times and encompassing weights and all that sort of stuff.

Justin (25:24) Yeah, that’s relatively fast though.

Jason (25:27) ⁓ So.

Okay, I would guess, I mean, it’s gonna be out this fall.

The question is, is it going to be out?

Justin (25:42) right before.

Jason (25:44) you know, winter break or can we get it out sometime in November? And I think that really falls on my shoulders. And the problem is, is that I have a number of other things that I have to kind of finish up first. like I can just like get off this podcast and I’m like working on it. have, you know, whatever handful of other things that I have to have to kind of finish up. So that’s, that’s the big problem is that, you know, we’re being, we’re all being pulled in five different directions, actually 30 different directions at once.

Justin (26:13) What kind of other stuff are you? I know what kind of other stuff you’re working on, but just for people who are curious, like what, like.

Jason (26:20) What is this so important, Jason, that you can’t, why, why, why just get the machine learning? What’s whatever the stupid thing you’re doing, sir. Okay. Well, one of the big things is, ⁓ gravity. ⁓ so, so w w what people often want to do is they want to be able to have a little more control on when they learn certain things. They want to prioritize something. So.

Sometimes what will happen is we have somebody who’s a college student, and they’re like, hey, I’m doing this linear algebra course along with the one I’m taking in college. The professor isn’t very good. This course is not strong. this is, Math Academy is allowing me to learn this stuff like I need to. But we’re doing these topics and we’re gonna have a midterm in two weeks. I need to.

can I do that, right? You know, it’s not going to help me so much if I’m deaf to all the other stuff that we’re not covering. And so what we developed, you know, geez, what I mean, I two, three years, three years ago, I think we initially created this, where you can, we call it adding gravity to an individual topic, you know, up to like a handful, maybe two or three, I wouldn’t probably do more than that topics. And so you look at a knowledge graph and you go, there’s gravity on this, there’s gravity on this and gravity on that as if it was like some celestial

body of a significant mass and it just pulls you to it. it’s like, so you can get it done as soon as possible. Of course you have to finish the prerequisites along that learning path, but it will prioritize those so that you can get to that point as quickly as possible. You know, and it’s, it’s, it’s, it’s also kind of examples if, you know, if you had like a, ⁓ let’s say you had a teacher and you had, you know, 20 students in your class and your algebra

one class and she’s like, okay, look, you know, next Thursday, I want to do this ⁓ project on projectile motion. And so I need to, the students need to know roots of a quadratic and finding a vertex and, you know, a couple other things, you know, because of this project we’re going to do and assume and imagine that you had an, a, ⁓ imagine that you had a tutor next to each student and the tutor would be like, okay,

I can do that. Now some of the kids might already passed that point. A few kids are just one or two steps away. Maybe a handful are five, six, seven, eight steps away. And they’re like, okay, we can get everybody through the prerequisites and master the materials so that when they do the project and project on motion, they got the skills in place. And ⁓ so gravity allows, will allow a parent to influence and prioritize the learning trajectories of

their kids, teachers to prioritize the learning objectives of all the students in their class. And will allow adults to say, you know, I really just, I want to learn these things, you know, for whatever reason. Either I’m just super, I think it’s super interesting or this is something else that I’m doing that requires me to learn that. Anyway, so that’s gravity. And so I built, you you built the underlying code in the model that handles it makes this happen.

but there wasn’t a really easy way for people to view this and interact with it and set it. So I had to build a whole UI for that. So I’m almost done with that. I’m hoping to finish that up. I wouldn’t say today, but every time I say I’ll finish today, it’s never true. So let’s call it end of the weekend. And then the next big thing I have, probably the thing after that is we get a lot of customer support.

Justin (30:12) for

Jason (30:14) ⁓ that it’s just, if we had like some emails go out, had like a list of common questions grouped by specific things, can account access, whatever it would really cut down. think, people being confused about things and, and, and cut down on the customer support we have to handle. And so I need to do some things that Sandy asked me to put together in terms of like some help kind of a help system, as well as some proactive emails that just address this stuff. And then the third.

Big one is ⁓ exams. So we now have midterm and final exams, which are kind of going to finish up the next day or two. So just like their quizzes are automatically generated and you have like retakes and stuff, we’re doing the same things, midterms and final exams. But I need to make it possible for schools. So imagine if you had a, you know, you have a classroom of students and you’re like, okay, well,

Unlike quizzes, I don’t want the students taking the midterm or final at home, right? They need to be proctored, has to be sitting here in class. And so I need to be able to set as a teacher, I to say, well, I don’t want it to be available until Thursday because it’s Tuesday and tomorrow we have a field trip or there’s a speaker or something and I want it Thursday and I want it from 11, you know, 15 to 12, 15. That’s when our class period is, you know. ⁓

Which is sensible. It’s what a teacher wants. Okay, I can’t, I can’t just have kids taking midterms and finals that are part of the game, willing, nilly, unproctored, you know? And so, you know, that’s not hard to do, but it’s not trivial UI work because they have to go and view and edit and postpone and change. And when is my class default time? You know, I mean, there’s a lot of, all this stuff sounds really, you think you get into it you’re like, wow, there’s just like a ton of, ton of little things you got to handle.

And the fourth thing I just realized that I kind of have to do so we have. Yes, he thinks we’re talking about. All these things that have to do with machine learning, which is kind of our life, right? It’s just juggling like a near infinite. List of important things.

Justin (32:32) Yeah, the more you do, the more important things arise because important things beget more important things.

Jason (32:41) Well, there’s another important thing, but I don’t want to talk about it yet because I’m already way off topic. Bring it up. There’s another important thing that I have that’s a blocker for some other people on the team. So that’s the thing. A lot of times I’m right. I’m working on stuff just like you are that is a blocker. For the content team for Sandy for students for school for parent, mean, you know, we have all these different kind of constituents. Right?

that need that that depend on stuff that you and I build. Yeah, you know, and so is even though it’s like I was like.

Even though yesterday, was it, was it yesterday morning or day before, I was like, okay, we, gotta get the, ⁓ the exam. Cause we, cause we went through this whole big thing about exams and like, you know, when our exams, when is a student ready to get an exam? Like, you know, how much review before they finished topics and that we have a whole nother thing and then you’re working all this stuff. And I was like, all right, well, we need that like ASAP because, you know, this school has a bunch of students who, and then otherwise we have to go stuff manually and it’s a nightmare.

You know, I got it. And then you’re like, sorry, I got to jump on something for Sandy because a student. Had this edge case of leaf nodes not being that whole thing. And that was and I was like, all right, you know, it’s like it was, you know, so that’s that’s the kind of stuff that happens. But. About I realize I’m terrible. I get a call. I get on all these digressions, so. But. We both, I mean, the math cat, I mean, the machine learning course.

I think is going to be ridiculously cool. I mean, it is, but it has been a monster of a project, right? And I’m just an absolute monster.

Justin (34:34) Scope

has just, like most things, you do it because you think it’s within close reach and then you start reaching. It’s with the grass. It’s It’s right there. Yeah. Yeah. I think when we started it, we weren’t even thinking about projects at the time. We’re just thinking about normal topics. That was even…

Jason (34:43) Then you’re

Justin (35:01) a challenge because like so many machine learning courses out there, like there’s just, there’s not like a really like, you know, you pick open a, an algebra one textbook or a calculus textbook. It’s pretty agreed upon what’s going to be in there. What kind of standards you’re hitting.

Jason (35:15) What is your differential equations course?

Justin (35:17) Yeah.

But machine learning, it’s just like, there’s so many places do it so many different ways. You just gotta like make sense of like what, what even is standard machine learning? I, you know, I think our machine learning course is going to be, ⁓ one, one of, one of the early standard, like sort of like what, just a, a, a, a well-researched take on what, what even constitutes

of a first course in machine learning. What even is that? ⁓ So anyway, just figuring out what that was. But yeah, later on, like you said, ⁓ we realized we needed projects. with projects come not only the content development for projects, but then, well, if you’re going to implement the projects, you need code. Uh-oh, we don’t have, we’ve not done any code stuff up until this point. Coding questions, entirely new, entirely new infrastructure.

How do you deal with this? ⁓

Jason (36:20) Yeah, I grades on and design XP and. All that stuff. Well, you know, and also with with projects, it was like, you know, the way we had done projects before it was like. You could complete the course without necessarily completing the projects and you could kind of you had you penalize in the XP, but we’re like.

That’s not acceptable. we, first of all, we need to change that, but absolutely. Like you could say, okay, well, you did a course in, ⁓ algebra two, but you didn’t do some projects, but you’ve mastered all the skills. You’re like, well, I mean, they’ve mastered all the skills. So they complete the course. You could kind of squint and say, that’s probably okay. Not ideal. Ideal is that you’ve done a bunch of application oriented projects that you really pulled these skills and put them into practice. But.

You know that most schools don’t most classes don’t actually do a lot of projects because this is just a ton of work and it’s very time consuming. And not. Not really essential. It’s just a it’s just like a really cool nice to have. ⁓ But in a machine learning course you’re doing machine learning you don’t do any projects. It’s like hey I can like I can prove I can prove this math or I can derive this thing or can solve some problem with gradient descent or whatever but I have no idea how to.

using this to solve anything that is remotely realistic. That’s just not a good course. so, and of course, we don’t want to make something that’s not a good one, we make an amazing course. We want to make a course like this is the new standard. And so that’s why we created all these projects. then, you had to go in and be like, wait a minute, you know, because I was like, wait a minute, is this part of, can they complete the course out of the projects? And you’re like, well, in current infrastructure, yeah. I’m like, that’s…

That’s not gonna work.

Justin (38:09) Yeah. Yeah. Yeah. And then that was a huge model of the, luckily that’s over, but yeah, it was a huge model update because we had to generalize our conception of what is an atomic unit of a course. Previously, it was just a topic. A topic is all there is in a course. Oh, and then there’s these, these multi-steps, these projects that are kind of like off the side that are kind of supplemental. Like you just get served one every 120, 150 XP, however much. then they just.

come and go. But now for the machine learning course and for a lot of other courses, it’s going to be these projects are tightly integrated. They are just as an atomic unit as topics in the sense that you have to pass every project. It’s factored into your completion, your progress in the course. Yeah. Yeah.

Jason (39:02) I mean, how long do you remember? How long did that take you? Because I remember that being a sizable.

Justin (39:07) Yeah, yeah.

Jason (39:08) your

part. This wasn’t like three days of work or something. This is like

solid like a really hard month.

Justin (39:17) Here’s the thing. sounds, I didn’t realize what I was in for until I started getting into some of the weeds. And just to say some of the weeds, because I know this sounds like so silly, like, oh, it’s not just a simple thing. Just say the projects are incorporated into the progress. Like, what’s the big deal? Like, well, okay, the problem is, okay, on the surface, yeah, okay, there’s a couple areas where we got to incorporate projects into.

progress and into ⁓ task selection, whatever. But the trickier ⁓ part is that the projects have to be incorporated as nodes in the knowledge graph that are actually connected up to everything and that play nice with the graph traversals. And by the way, all of these graph traversals, when they’re checking whether or not they have visited a node,

Already, they look at the nodes ID. Topics have IDs. Projects also have IDs. And you can have a project ID, 100.

Jason (40:27) know, it’s 451 and then I get a…

Justin (40:30) Same one in the topic. So I had to introduce this conception of like unique IDs, where it’s like not only the numerical ID record in the database, but actually the name of the table. So you’ve got like topic dash ID number and project dash ID number. anyway, so, but then that has to be incorporated like everywhere throughout the code. ⁓

And not just on like the surface level code, but at the most fundamental code that if there is one issue, like one like thing that I, I write, ⁓ I keep as UI, ⁓ as ID instead of writing ⁓ UID or something like that, just a small thing, ⁓ that could just take down the entire task processor because some fundamental graph traversal is not working as intended. So that was, ⁓ that was, that was not.

particularly fun, but I’m glad to be beyond that part. ⁓

Jason (41:29) like a lot of things, a lot of things we do. We’ll finish what you were saying.

Justin (41:34) Oh yeah, I was just gonna say, it’s like, well, this is probably what you were gonna say also, but it’s like, it, but this, this was like the zero to one step of generalizing our, our model and everything from just topics to also other types of, of, tasks, like projects and in the future, automaticity practice, automaticity practice is not necessarily like a topic. It’ll be some other kind of like.

flash card style, it’s just a different entity. And now the model is able to handle these different entities, reason about them in the graph, even.

Jason (42:11) That’s

why it’s so helpful. That’s why it’s painful, but sometimes it’s good to do sort of a breadth first search instead of just like, hey, we’ve got some of the work, so let’s just build on that. We’re like, let’s go tackle these other things that are really, really different. That’d be a ton of work, but it forces us to generalize, to solve those concrete problems and then generalize it and pull it into the model, pull it into the UI, pull it into the user, the student experience, and so it all makes sense because.

If you go too far down the line and you have lots of users and lots of customers and your stuff, you just can’t break things again and go back. It’s it’s hardened. Like that’s where you put the road, that’s where the road is, right? Well, it’s like, well, it goes like, it’s too hard to like, you know, we’re gonna create a new highway through these, you know, through this subdivision. It’s a nightmare. So you just kind of try and figure that out early. the way you do that is you say, well,

here are all these things that we think we might want to do, or we do think we know we want to do, they’re different, it’s going to be painful, but let’s just bite the bullet and do it now. And it slows down visible progress, so perceivable velocity, product velocity, but like, why is it taking so long? like, cause we’re tackling all these really hard, massive projects that is going to pay off and we’re going to be able to release all these things. But. ⁓

Justin (43:34) playing the long game. We’re planning to be around for the foreseeable future. And as a result of that, we are ⁓ just going full speed in all directions that we think is worth pursuing. ⁓

Jason (43:36) Play the long game.

Right,

so it’s definitely a breadth first, which means you get a lot of things that are like between 60 and 90 % done. They’re just sitting there and you’re like, God, we just can’t finish with these other things because you do have emergency things going on. You do have bugs, scalability problems. have, you know, just like really important features have to roll out for a user segment. You know, like we, you know, the schools have been coming on and wanting all this stuff. It’s like, know, there’s so much stuff. You just can’t.

You can’t just blow it off. You gotta deal with it. And then it’s like, well, why is the machine learning course down or why isn’t this done? It’s like, you know, yeah. Well, anyway, so I guess we could finish it. We can, I mean, there’s so much more we could talk about in machine learning course. I think we should get on to some other questions. So we’ll finish here and say, okay, the goal, we can’t make promises. The goal is to get it done this fall. It’s already recording this on Saturday, October 18th.

Which is shocking. I still feel like it’s like beginning of September, so I don’t know how the hell we got to middle of October. So. Which means that’s two and a half months less than two and a half months to get it done. So that’s a lot of pressure, but we gotta try and we started this last November. Right, that’s when we started. All right. All right, so we’ll we’ll go.

Justin (45:08) So the computer science course, then that’ll be after machine learning because that just depends on so much foundational infrastructure. We can kind of lump that into the same bucket of like,

Jason (45:21) Talk about

that later. That’s another discussion. So let’s bring that up as another, when’s the CS course? Let’s talk about that. We can talk about that a little bit because that’s a whole nother discussion. But yeah, that’s a thing. But I think for now. Yeah, what I was gonna say is just we’ll keep, we’ll address this again. We’ll talk in more detail. sure people might have some more questions just based on what we said here. But I think that’s more than enough for now about the machine learning course.

Justin (45:34) now.

Let me just end on one thing for the machine learning course. Just to convince everyone, it’s going to be worth the wait. Let me just read out a couple of the other projects. Because I only got through like four, but these projects are going to be so bad ass. There’s predicting supersonic aircraft noise with polynomial regression and regularization, evaluating satellite collision risk models with binary classification metrics, classifying exoplanet atmospheres with multinomial logistic regression.

Detecting art forgeries with K nearest neighbors classification. Discovering lost civilizations with K means clustering. Predicting hurricane evacuation traffic with decision trees and bagging. Managing injury risk in professional basketball with support vector machines. Classifying wastewater samples for pathogen surveillance with decision trees and bagging. Predicting soccer penalty shootouts with gradient boosting and decision trees.

Modeling electric vehicle range with multiple aggression and data transformation, spotting micro earthquakes with binary logistic regression, predicting braking distance for a self-driving car with linear and quadratic aggression, catching credit card fraud with decision tree ensembles and add a boost, detecting unrecognized snake bites with naive days. Like these, we have been working our ass off on this course and I know it’s been a long time coming, but it is going to be,

I’m a Schumacher.

Jason (47:16) something. It is.

And it’s only the first one. This is machine learning one. We’re at least going to have machine learning two because you had to push a number of topics from machine learning to machine learning two because the course was just we try and fit all university level courses into a semester length course because we want an apples to apples comparison. You know your differential equations course, your

Justin (47:22) Now

Jason (47:43) Indiscrete math, your multivariable, those are all like semester length, because if you don’t do that, people are just confused. like, well, is this equivalent or is this more or just like a whole year? So ⁓ this is supposed to be a semester long course. Now it is going to be a mouthful. I mean, this is like going to be a really hard semester long course, right? And do you have any idea how many topics there are in it?

Justin (48:06) Let me check.

Jason (48:08) That’s a lot, right? I mean, we try and keep it under 200 or something for a semester, of course, I think, right?

Justin (48:16) Yeah, well, it’s 140 topics currently. Okay. So, and that’s pretty stable.

Jason (48:23) That’s pretty

sweet. Yeah, it’s pretty stable. So we try to give you 120 and 180 is kind of kind of our sort of target. But this is a lot of projects which make it a lot a lot of work. and then but you had to push a lot of stuff to machine learning to that we were going to machine learning because it just wasn’t was just too much.

Justin (48:42) Yeah. Yeah. That’s just, For instance, ⁓ convolutional neural nets we originally had in machine learning one, but we just got to the point where it’s like, we, we had to push something, ⁓ some things out and, that was just one of the, it was one of those things on the border of like, well, okay. Machine learning, first course in machine learning has got to cover neural nets, but does it have to cover convolutional neural nets? Not really. That’s kind of like towards the edges. You can reasonably push that into machine learning.

too.

Jason (49:13) Yeah, so think that’s reasonable. You gotta, that’s the thing. You always gotta draw a line somewhere. When is the dividing line? There’s always a line, can do it the way you do it. like, well, if we never said that, we’d have 370 topics in a semester course. And people are like, wait a minute, this is not like a semester long. All right, it’s like some point, somebody has to draw lines. Okay, guys, guys, this is reasonable, right? Like this is good. And then there could be part two, right? Yeah, yeah, yeah. Nobody wants to watch a three hour movie, right? Like it’s, know.

Justin (49:40) There’s reinforcement learning, tons of reinforcement learning. But yeah, anyway, if we didn’t draw the line somewhere, we’d turn into one of those like 2000, like, you know, those 2000 page PDFs and all the math you need to know for machine learning. it’s just like ridiculous. like, it’s got like Zorn’s Lemma in it for machine learning. And you’re just like, what? Why is that in there? ⁓

Jason (50:02) Everything could possibly conceivably be it’s like an encyclopedia. It’s like the books I have that was the Cambridge, you know, encyclopedia of mathematics or something like that. Like, they’re like books like this thick and you know.

Justin (50:16) Yeah.

Well, why don’t we transition to the computer science course? ⁓ Just because that’s, I mean, that’s, very tangled up in the machine learning conceptually. ⁓ Like for instance, the, the reason why they, that even came into being was that we realized like, Hey, if we’re to have coding problems in these machine learning projects, like how do we ensure that students not only know ⁓ their math underlying these code, these problems, but also

have coding skills. ⁓ We don’t have coding skills on the platform. We’re just expecting students to come in, know how to work with functions and for loops and while loops and nested arrays and stuff. But that’s actually really, ⁓ you kind of need a really solid first course in coding to even be able to pull that off. So then that’s kind of how this coding course came into our.

field of vision.

Jason (51:16) Yeah. So

the thing about the, uh, the computer science course, which was interesting was I remember when you said you were, we were first kind of mapping out the machine learning course. And that’s something I asked you to do because Alex and his team were so swamped with finishing up probability statistics and discrete math. I recall. And as I got, could you just, I mean, you’re, you’re the expert on machine learning. So, um, you know,

give it a fresh shot, go to research, see what’s offered at the top institutions, what’s covered in the top textbooks, let’s get it kind of come into a consensus of things and you’re starting to working on it and you’re like, you’re gonna have to code to do this, and not just kind of code. you’re like, my concern is,

people are gonna come in and they’re going to struggle. And it’s not because there’s anything wrong with the content and it’s not because they like the math skills, it’s because they like the coding skills. And that’s gonna be a problem. Right? And I remember having that discussion came pretty early on, right?

Justin (52:36) And people aren’t going to know that they’re struggling due to lack of coding skills. And they’re going to think that there’s some other reason. Because it’s so hard as a student when you don’t know really what you don’t know. And yeah.

Jason (52:51) Why, you know why you know exactly the you’re at your ability to add to your attribution function is really high quality and you’re like, I don’t know. This doesn’t make any sense. I can’t do this. This is impossible. It’s like, well, problem is you don’t know how to do a list comprehension in Python, and that’s why this whole thing is confusing, right? Or you don’t really know the difference between an array in a I don’t know, a dictionary or something and you’re struggling with that with that, you know, and if you haven’t coded.

a lot then yeah, I why would you know when you’re gonna be confused, right? I mean, it’s so.

Justin (53:25) And like, even if you do know some of those fundamental, like arrays versus dictionaries, ⁓ what a for loop is, like, have you had enough practice actually like writing a for loop per some specifications that like stops at a certain value or what about ⁓ a while loop with an if statement inside of it? Like there’s lots of ways to combine these building blocks that you might take like a…

Intro to coding course and come out like, I know what a function is. know what a array is. know what a dictionary is. I can assign a value to, to, ⁓ to a space in a dictionary, but, ⁓ but you might not know how, how to put together these building blocks, these statements, while loops, et cetera, how to write helper functions, good helper functions, how to wrangle with some of the complexity, break it into modular components, that sort of stuff.

there’s quite a big gap between where most free online ⁓ coding ⁓ courses leave off versus the level that you have to be at to implement the stuff. It’s not an absurdly high level, but it’s just you need to be comfortable with some of this programming logic, not just the syntax.

Jason (54:39) Well, it’s what we talk a lot about is like, have to have a certain level of automaticity with it. It’s one thing it’s like, well, I listened to a lecture and I watched a video and I did a 15 minute project with dictionaries. It’s like, okay, well, that’s a great start, but you’re not, you have not even come close to reaching problem solving level automaticity with these skills. Right. And so.

And we picked Python because Python is sort of the lingua franca of academic coding classes, particularly in machine learning, right? mean, most people use Python. we’re like, we’ll focus on Python. But then one of the things that I remember kind of thinking about early and I brought it up with you is like, the reality is this

Some people are gonna view this as introduction to computer science with Python, which is what we’re really, what way we’re doing it. They’re gonna understand what that means, is introduced to computer science with Python. But other people are gonna read that as introduction to Python.

Right. And we’re going be no, no, it’s interesting. And then you’re going to have this ongoing debate about why you didn’t cover certain Pythonic ways of doing things. And you’re going to say, well, know, if they’re going to use Python for computer science, then they should know these things. It’s like, well, look, we got to get through the computer science skills and material. We can’t get sidetracked with all the cool ways of doing things in Python. It’s not, you know,

But I was like, I know that’s going be a losing battle. No matter how much we try and explain that they’re going to interpret it as intro to Python artists. I’m like, all right, screw it. Let’s make sure it’s both. It’s intro to computer science to Python, but it is a sufficiently comprehensive introduction to Python course. Right? mean, because one thing we always, it comes up is because once you’ve…

Once you’ve logged enough hours doing customer and technical support and you’ve suffered the pain of just this constant never ending stream of confusion, you’re always thinking, it’s like, how can I, how can we do this so that it can limit that? Right? don’t, I mean, you don’t, a, you’re trying to create the best possible experience for your, for your user or your customer, which is for them not to be confused, but also you’re trying to save yourself the pain of just

unending customer support requests and people are like, why aren’t you, why have you built this and this? It’s customer support all day. It’s like, well, why are you doing all customer support? It’s like, well, cause I thought if we just cut, you know, we just did do some things here, people would understand. And the answer is no, no, they’re not gonna understand. They’re just gonna be confused and they’re gonna email you and they’re gonna be annoyed and they may not always be polite. And it just ruins your day. Cause you get somebody who’s

So, okay, like how do we limit that as much as possible, head that off at the pass, limit the annoying to quitter discussion about Python and being a Python course or, you know, like just screw it. Let’s just solve it all. So that was part of the big part of the discussion that we kind of hit upon maybe a month after starting to work on the course, I think. I mean, there are other things. I mean, we started working on this a long time ago and we haven’t been doing as much of it lately.

Justin (58:10) Yeah, well, was kind of followed the same situation as machine learning with scope creep that was kind of necessary. Actually, because it was again, like, we’re like, okay, can’t we just like do the, the bare minimum foundations for what a student needs to know for these machine learning projects. And then we’re like, well, this is going to get compared to a first semester course in computer science. So we need that in right. And then that

Jason (58:35) said, I initially was just like a mini course, like this, like a mini, hey, like a mini Python, like just kind of getting started. But we realized that that even if we call whatever we call it, if people were going to be like, well, this is not, I don’t know, why doesn’t it cover this or it should cover that? I don’t know.

Justin (58:52) I thought Math Academy was comprehensive, like they didn’t even cover recursion in their computer science course.

Jason (59:00) What kind

of endless hacker news threads about that? You just see it coming, right? You just see it coming. so then we’re, that’s why, that was, cause that was even before the Python versus Intercomputer Science with Python course. That was the mini course versus, okay, this semester on course, right?

Justin (59:19) Yeah, that’s right. Exactly. Because right, you just know otherwise people are going to be like, well, Math Academy did a great job at math, but you know, for computer science, they are just is not quite it.

Jason (59:30) They’re just

out of their lane. It’s not what they do.

Justin (59:34) So we need our first foray into computer science to be something just that everybody can agree like, all right, high quality course. This is like the best intro to computer science, like that you’re ever gonna find. That’s the impression that we need to get.

Jason (59:49) That’s right. It’s kind of like if you’re like a famous singer and then you go to an acting role, people are going to be like, unless you’re really good, people are going be like, dude, just go back to singing. What are you doing? Or vice versa. The actors who try and put on album and it’s kind of like they really get a harsh treatment. People like to stick you in a lane. They don’t like you going outside your lane or at least you’re open for attack, which I guess is understandable. People think they’re good at something and they’re like, I’m good at everything. It’s like, OK, well, yeah, not.

Probably not. But for us, I mean, obviously we are professionals in software and we know how to do this stuff.

Justin (1:00:26) we’ve

taught it too, right? As part of the high school program, the URISCO program, which recently I wrote quite a bit out on ⁓ Twitter. ⁓

Jason (1:00:36) have you? Yeah, because we’ve been talking. OK, so explain. I think it’s just explain the history of your risk because this is really the foundation of both machine learning and the computer science course. Right. I’ll let you.

Justin (1:00:52) Yeah, yeah. Well, okay. So yeah, how did your skill even start? how started was, ⁓ I mean, your son Colby ⁓ was mathematically prepared for serious computer science. And in the school program, the computer science, like he was taking some coding course, right? And in eighth grade or seventh grade or something. And it was just, I remember he showed me his notebook one time. ⁓ because I asked like,

Have you written any code? Like any stuff you can show off? Like, and he showed me his notebook filled with written HTML. Not like a typed file, like literally like a pen on paper, write every character and like what, yeah, exactly. I was like, wait, what is this? Like, and yes, that was the kind of, that was what we were dealing with. So you were like.

And Colby was interested in computer science, right? So was like you being a software engineer, you’re like, wow, like my kid’s interested in software engineering. why is it that he is not getting a good opportunity to learn it? Like we need to make a difference for him and his friends too.

Jason (1:02:10) Which are? Yeah, let me jump in here. I’ll jump in one thing. So he had been complaining about his freshman year. So a friend of mine actually had funded this program in the school district. Great guy. Really successful entrepreneur. Really smart guy. And, you know, sometimes when things get built out for the school program. I mean, look, it was fine for the average kid, but not for math academy kids. And I had.

We had actually met with the people who ran this program ⁓ years before and I said, you know, you’re gonna have these math academy kids coming through who are gonna finish calculus in middle school and they are a different breed. They’re a different caliber. Not only are they all bright and motivated, but they know how to work and they know to figure, solve problems. So just get ready. And they’re like, yeah, yeah. And I’m like, they don’t get it. Like they just, don’t get it.

Like they’re dealing with kids who barely understand, they’re just learning algebra one and they’re just, and so I kind of knew even though we set this big meeting, it was like 20 people at the school district and we’re talking about, they were explaining what math academy is. And they’re like, that’s really cool. But they just didn’t get it. And I tried and I met with the, know, because I was friends with the funder of the program and he’s just, you know, very wealthy guy. So he just, had a lot of money they put into it, a lot of instructors. And he’s like,

You know, we, and I went and I was able to spend, you know, a few class periods sitting in the classes and kind of talking with their instructors. it’s a very, you know, and I liked the instructors. They were really bright and capable people. then one of them actually had worked at Google and these were not, you know, silly people. mean, they were some, they were doing the best they could, but they were, it was this calibrated towards the average kid. And, um,

I would, and Colby was even, he was in the honors section, but the honors section to me was like the regular section, and the regular section was like remedial. Right, the remedial kids were, had a struggle, they had, they would tell me they really struggled with basic logic, they really struggled with following simple directions, they even struggled with type, I mean, it’s like really hard time, and so. And then, and this is in ninth grade, right? And then the honors section was, maybe it was okay, but.

Call me like, dad, this is what I’m doing, this is dumb. And I’m like, all right, first week he’s complaining about it. And I was like, could just have a little patience, know, be like, don’t create a problem, you know, cause he had had a couple summers of like programming at like a Java, he did some Java courses, some little summer camp thing. And I did sing with him when he was like in fourth and fifth grade. So he had, he actually knew how to code to a degree and.

He was really impatient and I was like, well, what am going to do? Right. And I met with them and I tried to pair them, but nothing was happening. And so finally got to the end of, and it was their sophomore year and it was just, I, then I was lost patient. Like this is it. This is, this is just a waste. And this is when COVID started and you were sheltering in place with us. Now you are, how old were you when that happened? 23?

Justin (1:05:28) Let’s see, that was 2020. yeah, was about 23 or 24.

Jason (1:05:31) That was 19, right? 2019?

Yeah. Yeah. Yeah. So you’ve just been teaching for a couple of years and I had said, you because we were starting to work, you were starting to work with me on the math academy system, you know, on the model and that’s a whole other story. But when COVID happened and your then girlfriend, now wife was at Caltech and I said, look, we’re pulling the drawbridge up on Sunday. If you want to shelter with us, we got space.

We’re working together all the time, you know, fine. Otherwise, you know, that, know, and you’re like, and then when you found out that, you know, Sanjana was heading back to back home, they were shutting down Caltech. You’re like, all right, I’m in. Right. So you were so, so then we got to be like mad scientists, like 24 seven all day, every day. It’s all we did. And I, around that time, I was like, it was, it got into the summer. And I said, I don’t want Colby just sitting around doing nothing.

all summer. said, could you maybe, do you think maybe, because I, it’s really difficult to tutor your kids once they get past a certain age. It’s a little, it’s a little, ⁓ it’s, any parent who’s done too much of this knows how it can be fraught with a certain level of attention. And so I was like, could you maybe tutor him in coding, Python or whatever. And we would come up, we’ll do computer science and really go and you’re like, yeah, yeah.

And then I started feeling a guilty about it. And I said, you know what? Why don’t we ask the other kids in his math academy cohort, George and Dave and you know, guys and Eli, Eli, Riley and whatever, couple others. And I said, you can just see if they want to do it. I mean, they don’t have to do it. It’s COVID. Everybody’s just sitting at home. But I said, hey,

Justin (1:07:19) What else are you gonna do?

Jason (1:07:21) Justin is going to do that. Do you want to do this? We’re going to do like the math academy approach, but to math, computer science and do machine learning, you programming and machine learning or whatever. I don’t even know what we it machine learning back then, but, and they were like, yeah. And so then you, and then we sat around and we kind of like, we’re brainstorming, like, what can we teach them? And it was like, we were based it on the MIT introduction to computer science. It was like applied math and it was like computer science, was Python. And it was like machine learning. I don’t know. So that’s, that’s the setup.

And then you and I would sit there and brainstorm about it and you would put it into action. I don’t know, talk about, don’t you talk a little bit about.

Justin (1:07:55) Yeah, yeah. Well, it started out, I guess, like any computer science course covering the basics, covering, yeah, basic programming logic. I mean, these kids, I mean, it’s one thing to take a kid who is just a, who has a normal mathematical background as a 10th grader. don’t know. What do you learn? Like geometry, algebra two, that sort of level and have them program. These kids, like they had

like linear algebra, multivariable calculus, plenty of differential equations under their belt.

Jason (1:08:29) End of sophomore year, because of freshman year they did linear algebra and differential equations. No, sorry. Freshman year they did linear algebra and multiple-prepa calculus. In sophomore year they did differential equations and abstract algebra. They had done that. into that summer.

Justin (1:08:44) Yeah, exactly. Right. So we just like, I don’t have to tell, I don’t have to teach what a, what a derivative is or like, like they got their math in place and as a result of having the math in place, they also just get programming a lot better. Like what’s a function like they get it. Trust me, they get it. What’s, what’s a function of a function. They have no, no issues like grappling with the stuff in their head. Anyway, so they, they made tons and tons of progress. It was, ⁓ it’s kind of shocking how quickly.

They were making progress. ⁓ we quickly got out of scope of MIT’s Intro to Computer Science that we were originally craving.

Jason (1:09:23) red right like you’re gonna teach what

Justin (1:09:25) Yeah. Yeah. And then so we were like, well, what else are we going to do? What else? What else do we teach it? I think they got their fundamental programming, what are we going to do? And then we’re like, well, you I think you had this, this crazy idea that ended up turning out realistic, but it sounded kind of crazy at the time. ⁓

Jason (1:09:44) They

often do, right? As they often do. Yeah. Oh, they sound crazy.

Justin (1:09:47) I think you were like, Hey, Justin, like, there’s this, this, this research program called blondie 24. You ever heard of it? I was like, no. I, I had to look this thing up, but it was, it was about, ⁓ this guy, David Fogel, ⁓ using, neural nets trained through, ⁓ a genetic algorithm, ⁓ to play games, like started out with, Tic-Tac-Toe ended up with,

like checkers, a convolutional net setting. so these were research papers back in, I think the 90s or so. ⁓ Anyway, so you were like, why don’t you have the kids like implement like, like machine learning and like just build up to the blondie 24. Like, I mean, they already learned like the MIT Intro to Computer Science. Like that’s an incredible rate of progress. hard can it be? Yeah, exactly. And so, yeah, so I was like, okay, ⁓ I mean,

We’ll see, we’ll see.

Jason (1:10:46) You probably have hundred conversations like, why are you this? You’re like, okay. ⁓

Justin (1:10:54) I

mean, I was like, well, I mean, it’s not-

Jason (1:10:56) Say

it, it sounds like it’s not too hard, it’s easy, mean, but…

Justin (1:11:00) Something tells Yeah, it’s one of those things where it’s like, oh, that sounds really cool, but it sounds really hard, but it doesn’t sound impossible. It sounds, I mean, technically, fine, technically it’s possible. Okay, fine. If it’s technically possible, then I guess I have to do it because I can’t not do it. It’s just too cool. Like we can’t not do it anymore. thanks a lot, Jason. You put this like, dangled this in front of me. Now I can’t think about anything else. And it’s like just the…

It’s out of my mind. So yeah, so that was what we did. we just built up their machine learning chops. Obviously the research papers, that was such a high goal that we kind of had to do a frame shift of like, okay, we’re not gonna think too hard about that for now, but the kids, they’re gonna learn neural nets. They’re gonna build neural nets from scratch, back prop from scratch. Libraries, no, no, no. You’re not just gonna import like TensorFlow.

some Keras model and then like train that in like a couple lines of code and then say you you reimplemented a research paper like no no no you’re gonna build all this up from scratch.

Jason (1:12:10) Not only that, they would do the math first on paper. they’d have problem sets. The first step was to learn the math behind this, be able to solve math problems. So that’s in their brain. And then writing code, it’s like, I already understand this because I’ve solved them. I’ve solved gradient descent problems, you know, numerically by hand, and now I’m writing some code to do it.

Justin (1:12:32) Exactly. And so that’s kind of where, um, some of this ideas of like minimum viable project came into play. Cause I was like, I was like, okay, I got to get these kids to understand back propagation. And what I’m not going to do is I’m not going to just like quickly gloss over the math and then ask them to code it up in the context of a project because that just, it’s not efficient at all. And they don’t get many reps on it. And they just barely struggle through with the help of, um,

me and each other. And then they write it in code and then they never think about it again. And they effectively haven’t really learned it. So I was like, okay, we’re gonna make sure they know their fundamental skills and we’re gonna compress these problems down into bite-sized things, ⁓ apply back propagation, work it out by hand in a very small network, very small data set with nice numbers.

So you’re not like dragging around like tons of decimal places everywhere. ⁓ but yeah, and, and yeah, compressing compressing these otherwise, ⁓ computationally, just way too intense problems down to a size that you can actually work out by hand, like, like a math problem. And so that’s been actually the inspiration for a lot of these, ⁓ machine learning topics in our machine learning course. ⁓ you work out everything by hand before you go on to code stuff up in projects.

But it’s not like, we’re not asking you to do ⁓ like an hour’s worth of back prop computations for some.

Jason (1:14:06) Well, that was something learned even from calculus. Multi-variable calculus is easy to make a multi-variable calculus problem take 15 minutes. Like how do we turn it into a three minute problem? It’s like, okay, we scaffold at a certain level, let you focus on the part that is really the most challenging, most relevant to what you’re doing. Well, a couple of things that we’re pointing out. I remember one of the things we said early is import nothing. You build everything from scratch, including our library. So you had them build their linear algebra library.

So they had to start with a matrix class that could pseudo inverses and inverses and write all that stuff.

Justin (1:14:44) Yeah, reduce the echelon form, different types of determinant calculations. Yeah.

Jason (1:14:50) cofactor expansions and stuff. Like some of the earliest stuff that you did because it’s like, okay, and it was non-trivial, right? I mean, that stuff is not, especially for early coders, this stuff is not easy. But once they built their own classes, then it was like, okay, now we’re doing polynomial regression, and which is just an application of some matrix operations, some data sets, whatever. know, Colby still,

So Colby is a senior in college doing double majoring math, computer science. He’s in there for, he’s also in the combined masters for computer science or four plus one thing. So, and he still is like, yeah, I don’t even have to, I’ve already wrote all that code back in Urisco. He’s like, he’s machine learning classes. And he’s like, I just going to use my, you know, my, my linear algebra library and stuff. So he’s still, I asked him, said, do you, have you learned?

Anything new in computer science that you didn’t learn and with your risk of adjusting. He’s like, not much. No, not really. I mean, he a little bit with his systems class, you know, C and kernel stuff, but because we don’t really cover that, but machine learning that the data structures and algorithms stuff, you know, you had them do all the correct tree and graph traversals and breadth first and depth first. And, you know, all that’s you needed all that.

I mean, because I’m always I’m like an everything guy. I’m like, you’re like, should we do this? I’m like, I’m sure do it all. Every you’re like, Jesus Christ. I’m like, do it all. You know, Jason, there’s only so many hours a day. We’ll make more hours. Let’s go.

Justin (1:16:30) That was great though, because we covered, right, it wasn’t just machine learning. mean, it eventually got to the point where we were like, okay, they’re making great progress on machine learning. Let’s keep that going. But let’s also throw in like, I want to just, why not do like data structures and algorithms as well? Like, yeah, graph traversals, ⁓ Dijkstra’s algorithm, like all these sorts of. ⁓

Jason (1:16:53) They wrote their own genetic outcomes library. They even did their own reinforcement learning stuff.

Justin (1:16:57) Yeah, they did a little bit of that. So I should say for just to be clear, this was not all in that one summer before sophomore year. We haven’t gotten to this part of the story yet, but this turned into in-school classes that ran.

Jason (1:17:12) This was that summer and the following two academic years.

Justin (1:17:15) And we did end up getting to the blondie 24 papers as the nice cherry on top.

Jason (1:17:20) Well, in you, okay, one thing that you’re really good at is you is writing stuff down and say, okay, I hear all this stuff. I’m going to create documents. I’m to work, you know, so I don’t have to explain this again. Pick PDFs. for the for the next week, because then what we did is we offered it. I said, you know, because I once I realized how how well this was going, I contacted the school district, because, you know, we were doing math academy in the school district. So I had my contacts and I said, listen.

We got to keep this. This is lightning in a bottle. Can we do this advanced computer science thing? And amazingly, I was able to make it happen. And so when they were in 11th grade, it was just them. But then the following year, or no, did they start out? We brought in 10th graders that year as well. So we brought in 10th graders. So we had initial cohort, initially that summer. So they did like essentially like almost like a year worth of material or something approximating that.

Justin (1:18:11) Yes.

Jason (1:18:19) And so they were starting with the second year stuff as juniors. you had the stuff that you were doing the summer, you made that sort of what the 10th graders started doing. So we brought in a bunch of kids. We, I sent an email out to all the parents of all the kids in 10th, in the 10th grade, 10th graders, the math Academy program at that Pasadena high school. And I don’t know, was 15 or 17 people or 19 wanted to do it. It got whittled down. Cause what we realized is that math Academy.

is probably, you in the middle school and up is like the top 7%. It gets to the high school, it’s more like the top three to, I mean, ninth grade, it whittles down to the top three to 5%, but we realized that really it was about the only top 1 % could do this.

Justin (1:19:06) Yeah, that was definitely a high.

Jason (1:19:07) We

level it up a whole nother. This was a whole nother level. So you take about kids who can do a calc, who could be successful and get a four or five on the AP calculus BC exam in eighth grade. So we made it to a level that a lot of those kids were just weren’t able to do. And it wasn’t like for purpose. I wasn’t like, I was like, let’s make it so hard that people can’t do it. It’s just, we were just doing it with these kids or you were doing it. And it was just, we realized that it was.

As kids were just, some of the 10th graders were just not able to do it. It was just too hard. It was conceptually too hard, right? ⁓

Justin (1:19:43) Yeah, yeah.

okay. Part of it was like, okay, it was, was very conceptually challenging, but also the other part was like, none of this was on the math academy system, right? This was all, I was teaching it manually. So this was like the first, like, these are our test dummies. They’re they’re getting hit left and right by like, oh, I forgot to teach them this other thing beforehand that would have been really helpful. You hit. Yeah.

Jason (1:20:07) and

Justin (1:20:11) I am doing my best, there’s only, there’s only humans are limited by their, by their bandwidth to just keep all this stuff straight, organized layout. I’m building kind of like the knowledge graph in my head, putting stuff on the math Academy system, laying out this knowledge graph with plenty and plenty of practice, ⁓ adapting to each student. really, it, it, ⁓ it makes it a lot easier for students to follow along with the material and, ⁓

really learn and understand it to the degree that they need to. And now, so, in our, students learning eighth calculus and eighth grade and taking this crazy like multivariable calculus, linear algebra, ninth grade, like they’re working on the math academy system. So that means a lot more students are able to make the

belief. It’s not as rough writing. Things are very well scaffolded. They’re getting tons and tons of practice opportunities on every little thing that they need to know to make it to the next level of math. So to learn each next concept. Whereas in the Urisco sequence, like, yeah, I’m not, this is just me in a classroom. There’s not even a standard curriculum for this stuff. I’m just pulling this stuff out of like our conversations and what should they learn next?

Jason (1:21:30) I gave you a stack of like 20 books off of Publishel’s on data structures and algorithms and intro to computer science and intro to statistical learning. I always said, here’s a bunch of books you can use as reference. And then you had some stuff we were looking at, yours and online, and you were just pulling stuff out and just like, well, let’s do this. I’m like, ooh, what about this? Ooh, what about this? You’re like, hey, okay, well, maybe later, right? But it was so much. It was like,

Justin (1:21:45) Yeah.

Yeah, I was teaching this stuff in real time and it’s only the third year, ⁓ third and final year of this program that I actually like organized this all into a proper textbook, which is still a textbook is still very rough writing compared to when we have stuff on the

Jason (1:22:10) But

you created all those PDFs and then you put it, you actually published it early to sort of self-published it on Amazon, right?

Justin (1:22:16) Yeah, yeah, that’s right. It’s called Introduction to Algorithms in Machine Learning.

Jason (1:22:21) Right, I’m just looking at it right here. Here, I’ll grab it. It’s all dirty. Yes, I can’t read it, but anyway, yeah. It’s really good. that was sort of a, yes, dirty. But that was sort of the first, well, in fact, Alex, when you handed over your map,

Justin (1:22:28) Sure.

Jason (1:22:50) your sketch of the machine learning course, Alex and his team were using this as a reference for a lot of the topics. At least, I mean, obviously we had to go in a lot more depth to build on the system, but he kept talking about how great this book is. He’s man, he’s just did a great job with this stuff. There’s nowhere else we could find it where anybody explains any of this stuff in any kind of real concrete detail.

Justin (1:23:15) Yeah, that was definitely one of the challenges of the teaching the RISC-A courses is that there’s just, there’s no great content for this stuff online. That’s actually work, having students work through step-by-step through the computations. ⁓ Now you might, there’s plenty of stuff online that discusses like decision trees, such as discusses neural nets and whatever, but the vast, vast majority is just a kind of this, this top-down view of either here’s,

Here’s the recipe. Here’s, here’s, there’s the steps to how, how you, you, you build this now go ahead and build it. Or they might just say like import library, run like, Yeah. And so there’s nothing, there’s, there’s really little that steps through the math beforehand. And on the rare occasion that you find a problem that does step through the math beforehand, it’s usually something just really, it’s usually geared more toward like machine learning theory. And it’s like.

Jason (1:23:55) Press play.

Justin (1:24:13) prove this aspect about the neural nets or decision trees. you’re like, you’re just like, can I just get a problem where I just like work through like just just back prop on a simple case so I understand it because like these symbols are not like I need some concrete.

Jason (1:24:26) ⁓

Intuition comes from repetition, right? I remember just the pot that phrase pot in my head when we were driving. So this is an October. Alex and I and our and his wife, Cremie and Sandy, we were we were at your wedding in South Bend, Indiana, and we were driving back to the O’Hare airport to fly back. You’re flying back to the UK. ⁓

And we’re talking about this stuff and I, I, he said something about people, cause people are like, ah, I don’t want to do so many problems. just want like a conceptual thing. I want an intuitive understanding. It’s like, it’s repetition. Intuition is the result of intuition. mean, sorry. Intuition is a result of repetition. You have to do reps. That’s where the intuition comes from. Me or anybody explaining something to you that feels intuitive. You don’t have the intuition. You have to earn intuition. Intuition is earned through pain.

through failure, through suffering, through trial and error. That’s where your intuition comes from. You don’t wanna suffer, but you don’t get intuition. so, and that’s something that we, you know, obviously you wanna limit the suffering, like let’s just do the things and go through repetitions and say, okay, trial, okay, okay, right, I made a mistake there, right, okay, right, I see how this works. You do enough of the reps, whether it’s shooting free throws or…

you know, or doing math problems or whatever, it’s like you have to, you have to get the reps in, you get the intuition. But then when you get the intuition, then you can really understand how this stuff works and you can actually solve challenging problems. Because until you have the intuition, it’s hard to really see your way through innovative solutions. There’s just nothing to work.

Justin (1:26:16) Exactly. Yeah, well, one, just to give one example to this concrete example ⁓ is exploding gradients in neural nets and vanishing gradients. I, because the students were working out backfrop by hand, they would see actually like every time they take a, derivatives of something, if that derivative is, ⁓ is, is, is, is much less than

than one and they’re compounding it over and over, multiplying over and over. It eventually like just dies, comes down to zero. Everything just comes down to zero. There’s like no signals lost. Meanwhile, if the derivative is always like much greater than one, then it’s exploding. that’s why certain activation functions work well or don’t work well. They were able to see that.

Not just at a top level of like, I use this activation function and the numbers came out weird. They were actually able to see the mechanics of what is happening to cause this weirdness. And they could tell right away, like, uh-oh, I just noticed that like this number multiplied by this other number, the quantity is increasing. When’s it going to stop increasing? ⁓ Never. So, yeah.

Jason (1:27:35) We have to have a dampening effect. We have to have the dampening, the magnitudes of these things as they start to get out and they’re compressing them. But what’s great is you scaffold them up and they work through these projects and they run into these problems and they suffer. Like, what are we supposed to do? You’re like, what do think we should do?

We have to wait in like, get all like, okay, well, there’s something here. But yeah, they like, so you kind of lead them down that path. That was one great project is you can kind of lead them down a path to where they kind of understand the necessity for doing certain things. Right.

Justin (1:28:12) Yeah, yeah, exactly. Yeah. And then as you, as, as you kind of go down that path with them, you, you, get a better sense of how to steer them towards developing the same intuitions through repetitions, but having there be less potholes that they fall into and, and, and just like hitting on the essential, ⁓ repetitions as efficiently as possible.

Jason (1:28:33) Yeah. Well, ⁓ you know, one thing I wanted to go, but yeah, so, so that, so the, ESCO, you did that, you taught that for three years? Three. So you had, yeah, and multiple classes and multiple groups of students. So you really got an, we had a range of sort of aptitudes and, I mean, obviously at the high end, but then you had like the

Justin (1:28:45) Yeah. 2020 to 23.

Jason (1:29:04) super quants and then you have students who could just barely kind of struggle their way through it. But that taught you where all the potholes were. It’s like, okay, this is rough. This is we got to really expand this out. We got to smooth this out. need more. We need more learning infrastructure here and all these different places. And, you like said, there was only so much you could do. And, you know, at the time we had thought we had talked a little bit about like, I’ll building it to the system, but that was just like.

Even for someone as crazy as me, that was unrealistic. We weren’t going to have time to do that. It’s like, okay, well, at some point. But I remember around this time, this is early days, the ⁓ university level courses that were taught in high school were not using the system. They were doing that stuff.

Justin (1:30:00) Yeah, that was just the middle school really.

Jason (1:30:02) Yeah.

And it was already, even with the math professor we had, who was really great, they were, the students were forgetting all the previous math they learned. The student, and I remember they were the students who were in, I think it was a sophomore level, thinking differential equations in linear algebra, different equations in abstract algebra. And you were, we would do tutoring. We would offer free tutoring.

Justin (1:30:31) Yeah, it’s Saturday TA sessions.

Jason (1:30:33) And

you did a lot of those. had another guy who helped out and you did a lot of those. And I remember one, you came back, you actually did it outside at Caltech. I think you just picked, think you remember. Yeah, well, everything was closed.

Justin (1:30:45) Yeah, the school, they forgot to leave the school open for us. So I all these kids and I was like, what are we going to do? I just, we just transported them back to Caltech. We just worked at the patio area.

Jason (1:30:59) Right by the books where the bookstore used to be or where the yeah. Okay. So, I remember right. Cause we’re like a ⁓ mile half a mile mile to Caltech. Right. ⁓ so. And you’re like really frustrated and dismayed by how much math the students had forgotten from their linear out from the previous ninth grade year when they did linear algebra and multivariable. Because I was pushing.

Justin (1:31:26) Yeah.

Jason (1:31:29) the instructor at the time, said, look, you gotta keep integrating this stuff. You gotta put in some of the more advanced calculus stuff from high school and you gotta put in the multivariable. They’re not gonna remember it. Of course, he’s heads down teaching group theory. Right? Like, he doesn’t wanna work, talk about Green’s theorem. Right? I mean, it’s like he’s human. It’s like, I can’t teach everything. And…

So he did it because it’s hard. then what he would do is he then he did, he kind of begrudging was like, all right, fine, Jason. so he, because I’m saying, and I put together this program, these instructors we hired hot, and we would kind of give them feedback. We’d have a weekly meeting, they really didn’t answer to us. So it was kind of a frustrating situation because I couldn’t necessarily make them do something, but I could just encourage them and say, Hey, you probably need to do this. And, um,

But again, it’s a really big ask. It’s like, hey, keep everything fresh from the previous years while you’re this stuff. And then what happened is it was like four or five months into their sophomore year. And then you throw some multivariable calculus linear algebra they hadn’t seen in six, seven, eight months. Well, guess what? It’s gone. I mean, it’s not completely gone, but it’s really, really rusty.

And so now they have homework problems and stuff. They’re like, I don’t remember how to do this. I don’t remember this at all. And you were like.

You know, this is, because there, you have to, because we hadn’t been doing any type of a distributed practice or space repetition, excavating it was a gargantuan task. But had they been doing some of that continuously, everything would be pretty fresh. ⁓ here’s some multivariable, here’s some linear algebra, here’s some advanced trig stuff, here’s whatever, you know, just throwing stuff out there. It’s just, and, and, and, and so that’s where like we got to get this.

university stuff in the system stat. Like this is gonna be a disaster because you’re gonna have these students go through high school and we’re gonna say they did linear algebra and they did abstract algebra and they did real analysis and our kids are so smart. And then they show up at the university and they forgot most of it. And they’re like, well, I thought they could even place that linear algebra course. Didn’t you say that in their freshman year? Well, they did.

But they spent three years not doing it. So just like the students who graduated from your university undergraduate math program and went on to not go to grad school and hadn’t done linear algebra, they don’t remember much linear algebra because they haven’t thought about it in three years. I mean, they could throw a few terms out there and they could solve some simple stuff, but it’s really, really gone to seed. so I was like, this is, I said, but they’re not held to account for this.

Right? The students that are going to grad school in math, they’re staying relatively fresh on this. They’re tutors in the tutoring department, so they’re tutoring freshmen and sophomore math majors, so they’re staying up to date on the real analysis and things, and they’re grading for some of the first and second year courses, and maybe they’re doing some work to, maybe they’re studying for the Putnam, maybe they’re studying for the GRE subject exam, so they’re staying fresh, because they’re planning on getting a PhD in math.

But the ones who are like, I’m going to be an investment banker or I’m going to go and write code or something. They’re just like, I mean, I’m not worried about it. And then it’s gone and nobody pays attention and even knows that they forgot everything. Because it’s not, we don’t have comprehensive exams at the end of universities like I think they did 70, 80 years ago. so, but when math academy kids come out of high school and we hate these big claims about stuff that they learned and they go to

whatever university and then they’re like wanna place out of stuff and they’re not placing out of even the sophomore level stuff, looks, doesn’t look great. And so that’s where like we we gotta get this stuff. I had a lot of stress from this.

Justin (1:35:51) Yeah, I remember that.

Jason (1:35:52) Because this was under our name and I was like, you know, this, you know, and I’m trying to get them, trying to get these instructors to do this, but they just didn’t want to, they didn’t know how to go about it. was just.

Justin (1:36:06) It’s like you had the responsibility and people are going to hold you accountable for this, but you had really little control to make sure it was implemented properly in the classrooms, unless it’s put on our system and they use the system.

Jason (1:36:21) Yeah, that was the exact, and that’s the worst situation you could be in is have responsibility, but not have control. You know, it’s like, you know, and that was really, really frustrating. but anyway, but that’s what we kind of learned is like, you know, when you go from one course to another, the system needs to do, continue to do a type of space repetition from all the previous material. Otherwise it’s just going to evaporate. Now, luckily most stuff,

At least in high school and early it builds directly on top of it. And it’s only some leaf nodes. That’s kind of like you never use again, but so it’s easy to do sort of implicit repetitions and keep that stuff fresh. Now, when you get to the university level, well, then stuff kind of spreads out. And so now you really get to do more explicit review to keep these things fresh. Now, of course, if you start, if you’re doing that the whole time, then you can just, then it’s not a whole lot of work and you’re doing the more advanced.

you you do, you know, five or six linear algebra problems a week, review problems, but they’re picked from completely different parts of it and they implicit, they’re implicit reviews of lower stuff because they’ll pull in all these constituent skills. but yeah, that was just, I like, I was so exasperating, so frustrating. ⁓

But we learned all that. The thing is that we learned everything the hard way. We suffered in this way. I just remember, I remember you telling that one student said something. He’s like, well, after I learned something, I just forget it. I just delete it from my memory. You’re like, what are you talking about? And he’s just like your typical 15 year old kid. You’re like,

Justin (1:38:10) Yeah.

Jason (1:38:18) You know, of course, if you have a school educational progression that doesn’t hold you account for that, then you can do that. And then what’s the point of it anyway? If you’re just going to forget everything you learned, then what’s the point? The point is not to learn and then forget. The point is to retain, at least retain a sizable chunk of it. And the stuff that you don’t retain, you can do just a little bit of work and get it back. Right? It’s like…

It’s like lifting weights. If you lifted weights and then you took for like two or three years and then you took six months off and you lost everything that you learned, all the strength and muscle or whatever, and it would take you three years to get back to work, it’d be like endlessly frustrating. Now luckily, you take six months off, within a couple months, you’re mostly back.

where it took three years to get, not that big of a Even if you stop working out and you’re like, don’t know, this is like I would do, I’d lift weight, then I’d spend all the time playing soccer. And it’s like, well, I’m not quite as muscular, quite as strong. Couple months, I’m right back. I’m almost near my max. And it’s like not a big deal. And that’s the same way with learning. It’s like you want them to learn it in a way that it’s like, okay, you’re not gonna stay razor sharp on every single thing.

if you stop doing it, that’s completely unrealistic. The human brain just doesn’t work that way. But what we can do is, is it going to do laser razor sharp on stuff that’s a little further down. That’s been, you’ve had a lot of repetitions on it has been sort of the substrate on which the more advanced stuff, but the more advanced stuff you can pull back without too much work. You don’t have to spend years getting it back. It’s like, well, I got to, you know, I just spent a couple of months and kind of get going again or whatever. Anyway.

That’s what the high school program was such a great, I mean, we can go on, there’s so much more to talk about, it’s taught, was such a great like experiment.

Justin (1:40:20) Yeah, yeah, totally. Right. And like you were talking about with the space repetition, it’s like the goal is like, it’s okay for memory to get a little fuzzy, but we’re trying to forget what we’re trying to prevent is from getting so decayed that you have to like relearn it from scratch. ⁓ and that’s what was happening to a lot of these, these kids in high school programs. Like at the TA sessions, I’d be like, well, I mean, you, how do you solve this problem? Well, do you remember how to take, how to, how to compute the, ⁓

the eigenvalues of a matrix and they’re like, no. like, yeah, yeah. And it got to the point where it’s like, I, even after like, I gave them a reminder, they would need kind of like many practice problems to get back up to speed. was, like having to teach them almost from, from, from scratch.

Jason (1:40:53) We haven’t done that in eight months or six months.

So they went down to like the 15, 20 % memory level instead of like the 85 % where they’re like, oh, right. Yeah, right. Okay. You know what mean? It’s like, I don’t.

Justin (1:41:20) Yeah. ⁓

Jason (1:41:21) So

inefficient. just can’t, you can’t build on it. It’s just, it’s just, you’re almost like, forget it. It’s not even. And I think that’s what happened with some of the kids. We were doing the Matt, the risk. they just forgot too much of the foundational linear algebra and cow and multivariable.

Justin (1:41:40) Yeah, I remember the first Eurisco cohort actually did struggle with partial derivatives because it had just been so long since they had done that sort of stuff. Because they weren’t doing this on the system, right? It was just up to a single teacher’s ability to manage spaced repetition across all the top. Yeah, which they weren’t. Yeah. Yep. And so they had forgotten like…

Jason (1:42:01) Thanks.

Justin (1:42:08) Tons of derivative rules, even stuff like the product rule, the chain rule. Yeah. Okay. Yeah. Yeah. I remember this. was teaching them back propagation and then just saying like, okay, so what do we do after this? You would do the chain rule and then like just getting some blank faces. I’m like, come on guys. It’s a chain rule. Yeah. But they forgot the chain rule and yeah.

Jason (1:42:29) Pain wall, right? ⁓

a single variable or like a multivariable like chain.

Justin (1:42:38) Well, okay, so the chain rule with single variable was luckily in a better memory state. They didn’t only decayed maybe down to like, ⁓ I don’t know, maybe 60, 50, yeah, yeah, something like that. But with multivariable, yeah, that had actually decayed even more because they had…

They want a shorter space repetition interval for that. Right. was, was technically more recent, but they hadn’t built up enough practice to get into the long term space repetition where your intervals are like half a year or more. Or it’s like, that’s kind of what it was with the, with the single variable chain rule, but yeah, multivariable chain rule that I basically had to. To reteach that.

Jason (1:43:25) so frustrating. So frustrating. ⁓ so like, let’s, okay, I think we got, so let’s get back to the computer science course. So the state of the computer science course is that we’re going to, once the machine learning course is done, after the new year, we’ll reengage and start looking at that. Because we to get the machine learning course out first.

And the differential equations course, well, first we’re finishing up the SAT course, SAT prep course, that’s a whole nother discussion. Then the differential equations course, we’re trying to get that finished up. And then machine learning, get that one out, and abstract algebra. And that stuff is gonna take us through, well, abstract algebra will go through the new year. And then after that, sometime up to any year we can really.

sort of start thinking a little more seriously about the computer science course and.

you know, other things. it’s like, yeah, see the real challenge about this is that it’s very hard. You think like, like, development should be like embarrassingly parallel, which is not. That is true if Alex, if we could clone Alex and clone some of his senior lieutenants. The problem is,

that whenever we’ve tried to hire people, even very talented, skilled people, it takes a long time to really understand the pedagogy and how we do things and how this stuff’s supposed to work, how to write good questions, how to write good explanations. mean, it just takes frustratingly long to do that. so ⁓ Alex has to be very much in a loop on all the course development, along with the senior lead. He just has to.

It was just, whenever he kind of over-delegated and tried to, just starts spinning out, just got off the rails, right? Every time. And so to a certain degree, we’re limited by, he doesn’t have to develop everything, but he has to give everything feedback and ultimately at a certain level. mean, it goes through multiple levels of people. have a variety of levels of people in the content team, editors and…

course designers and whatever. But if Alex is not tightly involved in every, on the development of course is just not going to be at the level that we need it to be. And it’ll just be really extremely, extremely disappointing and frustrating. And we tried that a couple of times and we had to kind of pull things back. That happened machine learning course a little bit, right?

Justin (1:46:22) Yeah, mostly with the computer science course though back.

Jason (1:46:26) Computer

science really got off the rails. But even the SAT course a little bit, mean, some of these other things, this has happened numerous times and we’re just trying to go faster. And it’s, it’s, um,

It’s frustrating it is for anybody who’s waiting for some course to be ready, it’s even more frustrating for me. It’s like, let’s go, let’s get this done. But that said, SAT course, I mean, sorry, the SAT course, the computer science course is something that we will have to reengage with after the new year. ⁓

I agree it’s potentially an important course, a valuable course, and a lot of people waiting for it. But there’s a lot of courses and a lot of people waiting for it. The SAT course is a big one, right? And that’s been a monster, monster effort as well, like the machine. That’s a whole other discussion which we can get into.

Justin (1:47:28) Yeah.

Okay, rapid fire around. First, let’s briefly answer some questions about space repetition improvements. ⁓ So we’ve got a few questions about that and mostly, I’m not gonna go through all the questions, but the answer to encapsulates most of them is what are we planning on improvements to the space repetition system? There’s two main.

things that we we’ve talked about so far. ⁓ one of them is incorporating reference reliance into, ⁓ assessing how successful students performance is. Like there’s right, right now the, the, algorithms are based at least outside of the diagnostic. The algorithms are based on like your accuracy. ⁓ did you get it? Did you get the question right or wrong? But really there’s, there’s a whole gradation there.

Between like, did you get the question right after peeking back at the reference one time for 15 seconds? Did he get the question right without looking at the reference at all? Did he get the question right after you looked back at the reference, like a five times for a minute each, like there’s, there’s a, there’s a big difference. And so, so we, gotta, we gotta get that built into the, the space repetition system.

Jason (1:48:56) Yeah, mean, one thing we always talk about is what would a tutor do, right? Whenever we’re to be like, how should the system behave? So if you had a tutor sitting next to a student and you say, factor this quadratic and the student looks at it second and then sits down and goes, figures it out and does it no problem. You’d be like, And or if they do it really fast, they’re like, yeah, bang, bang, bang. You’re like, all right, yeah, okay, you don’t do it.

Anyway, well, let me try it with a negative number or a fraction that you’ve got a couple of various. Okay. Another one, you sit there and they’re like, thinking, You as a tutor be like, I don’t think they remember this so well. If they took a long time. Now, if they not only took a long time, but then they went back to their notes.

I looked at a textbook, you’d be, as a tutor, you’d be like, okay, well, clearly we need, the student needs more practice. They’re forgetting this stuff. And so then you would make a mental note, the student is not really strong on this material yet. And what you would do is you say, we need more practice probably now, and I need to move forward some practice that I was gonna do later, sooner, because this is starting to fall away. Now, and if you’re seeing,

that behavior on lots of things, you’re like, we need to slow things down. We are moving too fast and the student is becoming overwhelmed and they’re not able to remember the things they’ve learned. They need more practice. need, we need to shorten the repetition intervals. And so we need to take that. And in order to do that, now we do that already when they don’t do well on review, they struggle and get partial credit or no credit. Like that is incorporated into.

how well they know that topic as well as overall pace of learning. But what we haven’t been doing so far and what you’re saying is right is that this secondary sort of behavior, which is they are asking additional scaffolding.

Justin (1:50:59) Yeah.

Yeah, it’s just one of those things where it’s like our initial approach is, I mean, we just hit on the first order effects. Like, did they go to the question right or wrong? Cause that’s what you do when you’re building a product and you just have to solve the biggest problems first and then ⁓ later down the line, come back for second order improvements and things. So that’s a second order improvement in the space repetition system. ⁓ But also,

not just the space repetition system, but also just the whole kind of ⁓ pipeline as a student goes through a lesson coaching them on what to do. Like one thing that I have seen ⁓ students, even adult students do a lot is look back at the reference too often, like thinking that it’s free to look back at the reference. When in reality, if you are trying to recall something from your head and

And you look back at the reference instead of trying your best to lift that weight off of your, long-term memory. If you look back at the reference, you’re basically just letting the spotter lift the weight.

Jason (1:52:06) Yeah, so you’re weightlifting and you’re like, I can’t get it and the guy behind just lifts it up for you like, okay. He didn’t really lift the weight. I mean, most of it, but you needed help. And so therefore, if I’m your trainer, I need to lower the weight because you’re not quite ready for that. It’s gonna you’re gonna injure yourself. not gonna be productive.

Justin (1:52:11) Well, yeah.

Yeah, but also like you don’t want your spotter. Sometimes people will ask for the spotter’s help before they even get to the point of trying really hard. It’s like just the moment that it stops becoming super easy, they’re like, okay, look back at the example. When in reality, that is the moment when you are getting the most bang for buck out of recalling the information.

Jason (1:52:48) The struggle, the struggle, it’s the act, the struggle during the active recall process is when you are strengthening the memory, right? Or re-imprinting the memory, think is actually how it works. But ⁓ yeah, so that’s critical. I think, yeah, so okay, I don’t wanna go on and but that’s really what we need to build in is we already have this data. I mean, we track.

you know, the behavioral data so that we can do this kind of a thing because it’s important. Because you’re like, well, why, you you’re just like, well, how can we get this to behave as good as an expert tutor? I’m paying $150 an hour to sit right next to me who has a perfect memory. Well, then that tutor is taking into consideration a lot of more things than just where they got the question right.

Justin (1:53:47) Yeah. And the tutor is actually instructing the student, Hey, like, don’t look back at the reference right away. Like I want you to try to pull this from memory. So that’s what we need to also do. Yeah. There’s an active, there’s two, yeah, right. There’s, two, the two sides. So one side is like just getting those second order and third order interpretations of what is actually happening with the student looking back at the reference. But, ⁓ yeah. In addition to that, it’s just like, well, try to.

Jason (1:53:56) Coach them.

Justin (1:54:17) coach the student not to actually look back at the reference unless they absolutely need to. Because that is a way that sometimes students kind of shoot themselves in the foot. ⁓

Jason (1:54:28) They

shortchange the learning process. They shortchange it, right?

Justin (1:54:31) Yeah. Or if they have like the, worked example up in another tab and then they’re looking at that while solving the problem. Like, is, that is the worst. Never, never do that. ⁓ cause yeah, it’s just misleading. ⁓

Jason (1:54:44) again, you’re cheating. Well, know, along some of our lines, I remember there was a friend of mine and he had his son using this system. And this was an earlier a few years ago. And he’s like, you know, Jason, like my son, he’s sick, it’s moving a little fast and he’s struggling now. And I’m like, let me show you question. go, are you helping him?

And he’s like, well, yeah, I sit with him. go, OK, are you allowing him to get the questions incorrect?

And this is a smart guy, went to MIT. This is not, he’s not a dummy. He’s like, well, no. go, okay, that’s it, Aaron. You gotta let him make mistakes because if you, if every time he’s about to get the wrong answer, well, are you sure? Maybe we should rethink that or maybe we need to, then what you’re doing is you’re, the information you’re giving the system is that the kid does not make a mistake and that the kid is mastered material with no effort, which means the system is gonna be like, hey,

I mean, if this is that easy for you, might as well go faster, because I don’t want to bore you. Right? And that’s what the system does. You are misinforming the system. And I know, as a parent, you’re not trying to do that. You’re just trying to help your son. And you know, kind of, I’ve done it too. I’m the guy, part of the system. I have done it. You know, I’ve said to my daughter, and Sandy’s like, okay, well, you know, we got dinner. Can you hurry up? Okay, all right. And then I’m just like, are you sure? You know, and.

You if she gets it wrong and she has to do another one, just got to, I mean, you really just got to like, them do it. You know? mean, you gotta, you gotta let them make mistakes because the system that’s, oh, that’s what the free region of the system is taking in.

Justin (1:56:34) Yeah. Yeah. Yeah. And it’s like, not only is this being incorporated into their repetition intervals, but also it’s only if they make a mistake, they’re going to get extra practice on it. If you’re not making mistakes, you’re going to get the minimum number of questions.

Jason (1:56:50) Yeah.

So whenever I hear from a parent like, I think the system moving too fast or even adults, it’s because you’re probably you’re misinforming the system you’re using either as a parent is helping the kid or they’ve had a tutor who’s sitting with the kid and doesn’t let the kid a mistake or as an adult, you’re like every time you get a question, you go back and review it in reference mode. You know, because you don’t want to make a mistake because you’re a perfectionist and you want to everything right. I’m a perfectionist too. I get it. But that’s

You’re, you know, and so that’s why we have to basically the system monitor that. Is best we can now for students for for students or adults who write it down and there’s red fritzing a piece of paper. mean, we don’t have a camera on you. We can’t like say, hey, you’re looking your notebook. It won’t be perfect. You know, and all we can do is say, don’t try not to do that. Try not only look in your notes after you’ve gotten something wrong.

and you’re trying to remember why you got it wrong. If the explanation isn’t enough and you want to go to your notes, but try to resist the urge. But hey, we don’t explain that really to anybody. So it’s easy for people to go, well, I thought it would be good to look at my notes and I thought it would be good to help my kid. like not in the way this is happening.

Justin (1:58:05) Yeah.

Yeah, so in summary, the two aspects are there are some improvements that we can make to the repetition, the space repetition system itself, incorporating more of this information in. But there’s also improvements that we need to make on just aligning people with the behavior, the proper learning behavior to give the system a good signal. two things to one, like get the most they can out of the space repetition, not

saying spotter lift the weight from me as soon as it looks like I start to strain a little bit. And then two also to just give the system an accurate ⁓ understanding of what’s actually going on.

Jason (1:58:47) And by the fourth thing I said I had to do before I got back to the machine learning was some in-lesson coaching. I have to do some work on that because it’s blocking some other people who working on that. that’s another, it’s a big one. an in-class, intra-task, we call it inter-task, know, information and analysis and all that stuff is like, there’s a ton of low-hanging fruit there for us. I think we can really.

take things up a notch or two, right?

Justin (1:59:19) Yep, totally. Another category of questions that’s kind of similar to here is about ⁓ sort of a more granular factual knowledge, things like trig identities and just tables of information ⁓ that ultimately amount to math facts. so, ⁓ okay, so a common question is like,

Should you be supplementing ⁓ the Math Academy system with like spaced repetition cards where you write down like math facts and then do spaced repetition on them? that there’s, ⁓ you should get a lot of this repetition practice from the problem solving itself, provided that you are actually ⁓ trying to recall this from memory, doing your best to recall this from memory.

Jason (2:00:02) I with you.

Justin (2:00:16) only peeking back at notes or reference material, just the tiniest peek ⁓ after you’ve strained as much as you can to remember and we’re not able to. ⁓

Jason (2:00:27) Strain

as much you remember doesn’t mean you sit there for 10 minutes. There’s a difference. It’s like, okay, so when someone says, I’m not sure I remember how to do something, it doesn’t mean strain for four seconds and then like, I don’t know, and go to the back of the book, right? It also doesn’t mean sit there for 10 or 15 or 20 minutes. I mean, it’s a little context dependent, but you just, you you could sit there for at least 30 seconds, right? And go, try to think about it, trying to pay like,

Justin (2:00:30) Yeah, yeah.

Jason (2:00:56) okay, how does that work? You know, whether it’s 20 seconds or 45 seconds, something in that range of reasonable, maybe a minute. Okay, it gives a minute goes by and they’re like, I really cannot remember this. Okay, fine. But don’t grind your, like I’ve had people, like we’ve had people in the diagnostic, had one guy who said he’s been working on the diagnostic for months. I was like, I would never have anticipated that. And he was going on Wikipedia and trying to look stuff up. And he’s like, that is totally.

not what a diagnostic is for. A diagnostic of what do you know now, not what you can go reteach yourself, right? It was so weird that he did that. But there have been other people, he’s not the only one, but there are other people who just interpreted in that way. it’s an interpretation I didn’t expect, but it happens, but it’s not helpful. It’s like be honest, put in an honest effort, but give…

accurate information to the system through your actions so that it can adapt to where you are right now.

Justin (2:02:05) And now at the same time though, is helpful to have more practice on these underlying math facts to kind of like bring them out of the problem solving context and just drill them more frequently. It’s kind of like if you’re in athletics, you don’t just exercise your skills by playing games or even by doing like

like one minute long drills. There are some skills that you just have to like be really, really solid on like shooting free throws. You don’t practice like shooting one free throw every, every couple minutes like during a game or something. No, you actually go to the line and practice doing that. That’s kind of like, ⁓ math facts, automaticity practice. And so we, do have, ⁓ plans to, get math facts, automaticity practice in the system. Not just, not just multiple.

This kind of started out, were thinking about like multiplication facts and stuff like that at the elementary grades. Like, what do we need to expand this knowledge graph even lower into the elementary grades? But we realized like, hey, we should really do this throughout all of math, like derivative rules, ⁓ sorts of table, trig identities, all sorts of these facts that come up and can be practiced rapid fire outside of an accurate.

Jason (2:03:31) Efficient

very efficient way because it doesn’t take long to blow through these

Justin (2:03:36) Exactly. Yeah. Yeah. So that’s very much on our radar.

Jason (2:03:40) Well, automaticity,

specifically math facts, is really high on our list. I mean, you did a bunch of work on this.

We had to stop, you had to work on some other stuff, but that’s clearly a hole in our learning platform because a lot of students come to us in pre-algebra and algebra and they don’t know their education tables. And they’re not very good at fractions. And that is very common. ⁓ Schools have been not doing a very good job of that.

and they’ve been sort of drank some crazy, ⁓ what do call it? Cool aid, yeah, some crazy cool aid. like, we don’t have to memorize anything more. It’s like.

Justin (2:04:34) Cool aid.

Yeah, that’s totally fall-

Jason (2:04:42) That’s a whole nother discussion. That’s a whole nother discussion. It’s totally wrong. It’s like, you don’t have to practice your free throws. You’ll just like, you just know it, you know, just go play basketball. It’s like, what are you talking about? It’s so dumb. But, you You know, and I hear from teachers and chief academic officers and tutors, and they’re just apoplectic about the situation because you get kids who are in sixth, seventh, eighth, tenth grade. Finger counting.

don’t know the multiplication tables, it’s like, how the heck are gonna factor quadratic when you don’t know the multiplication tables? Can’t do it, not really. And then guess what? Now we can’t factor quadratic, now can’t do algebra.

Justin (2:05:24) And if you do manage to grind through, ⁓ just figuring out these factors on the fly, it’s gonna take you way, way, way too long. It’s gonna take you like 10 minutes to factor.

Jason (2:05:34) Which some kids can do in their head and be like,

10 seconds. So you’re not going to want to do any proms. It’s through, yeah, teacher gave us four factoring problems and it took me all night, you know? And you’re just like, dude.

Justin (2:05:38) Yes.

When you get to calculus, if you’re taking 10 minutes to factor a quadratic, that’s only one component of a calculus optimization problem. You’re going to be spending half an hour on this problem that should take you two minutes.

Jason (2:05:58) Well, you ran into this problem. taught at a private school for one year. And for AP calculus and physics. Was I kind of a disaster, right?

Justin (2:06:03) You’re dead.

Yeah, yeah.

Yeah, I knew they were going to like come in forgetting some stuff, but I didn’t realize how bad it was going to be. Like I had to reteach logarithms, unit circle, not like unit circle, not just like your main like trig identities, like secant tangent stuff. mean, like, what is the sine of pi over three? Like they had just, they didn’t even, they didn’t know that. They, they, they,

Not only did they not have that memorized, but they didn’t know how to like draw up the unit circle and like figure that out on the fly. was just like, it was gone from their head. So.

Jason (2:06:45) You

me your physics class the kids couldn’t do basic algebra 1 skills.

Justin (2:06:50) Yeah,

so I should clarify what I was talking about just now was the AP Calculus AB class. The physics class, yeah, it was a struggle to solve linear equations. This was after they had taken at least one algebra class and linear equations as in like AX plus B equals C, like just really

Jason (2:07:12) simple,

really straightforward. You’re like, you’re, I remember you were trying to figure out like, what am I, what am I really supposed to do here? How did you solve that problem? Just real quick. We don’t have to get too into it, but how did you even?

Justin (2:07:15) Yeah.

Well, for the physics courses, mean, well, so yeah, the problem was, well, ⁓ Luckily the physics courses were not AP physics. They were just kind of whatever the instructor decides he wants to teach. So I mean, the, I could do was, was kind of limit the scope of the courses, not to require too much math. So there was a lot of, there was a lot of physics, that I was kind of hoping we might get to that we, we were not able to get to.

and, yeah, it just, it was, was a very limiting factor, very constrained what we were able to cover.

Jason (2:08:04) Yeah, I remember that. remember you were kind of in shock. And when we were first talking about this and how far behind and again, the previous, the, the, the previous, the thing is, is that the, happened is that the, the, the teachers had before had just neglected doing things that they were supposed to do. And then it just becomes a problem where the teachers after it just, it’s just too much to even clean up. Right. You do the best you can. It’s like, I can’t, I can’t teach algebra one.

Justin (2:08:07) Yeah.

Jason (2:08:33) and physics at the same time, especially when they’re at a school where they don’t really like you to sign homework, right?

Justin (2:08:40) Yeah. Yeah. And it was, oh yeah. I mean, that, is another thing. was a mastery based learning, which to their version of mastery based learning was not the same version as Benjamin Bloom learned the prerequisite first. Their version of mastery based learning was let’s, instead of assigning like, um, numerical grades on things, let’s break up the course into like 15, just, just list 15 things that they need to master in order to master this course. And then, and then give them a

like acceptable, like neutral or bad on each one. And, but the thing is like a lot of these, these, these mastery standards were not, were not having to do with the, with the, with the content so much. It was more of just like some kind of nebulous sorta is, is student is, is able to work.

is able to leverage like contextual information about the problem to set up a model. Like, well, like this, this, this kind of standard thing is like, um, it doesn’t say whether it’s like quadratic or linear equations or what, like there’s, can, it’s, it’s so easy to, I mean, if you give us a student, like a, not so great grade on it, it’s, it’s so easy for them to come back and be like, well, what do you mean? Like I solve this, this easy.

Jason (2:10:02) Well, how do they, why am I a student gonna be? you’re just.

Justin (2:10:06) Yeah, it’s it’s indefensible, basically. ⁓ Right, when the standards are that vague. And yeah, another thing was like the students can test as many times as they want to do the standards, which like, on a system like Math Academy, where you get to reattent lessons, like that works out OK. ⁓

But when there’s just a one teacher like manually creating all this work, it kind of turns into the students are not studying for the tests because they know they’re just going to have a reattempt afterward that can totally obliterate their first attempt. Like it doesn’t even matter what they do on the first attempt as long

Jason (2:10:55) You’re

just teaching him not to try and the teachers like fine. It’s the same problem. I just put in a slightly different values and the kid kind of learns just to do that. It was like I can’t. This is your fifth attempt. I whatever you know.

Justin (2:11:04) Yeah, right.

It’s like, yeah, why would you do the effort to study initially when you can just wait for the exam to kind of telegraph?

Jason (2:11:14) I’ll yeah, I’ll take the first exam and then I won’t worry about it. And I’ll kind of have a sense of it. And look at the second, Tim, and the teacher will make that sort of similar. But by the third or fourth, the teacher just gives up and just makes it the same exam with maybe some different numbers or maybe even the same numbers. just like, yeah. Anyway. Okay. So let’s, let’s, let’s, let’s move on. Um, what’s the next, what’s the next.

Justin (2:11:37) Moving on. So ⁓ another thing ⁓ that somebody asked was, what is your vision for incorporating projects into the math curriculum at the early stage? Like, for instance, like algebra and geometry, and not just the machine learning course, but all throughout the high school and even elementary math. Have you considered interspersing coding or optional coding projects throughout the curriculum?

starting with algebra or any particular level? The answer is yes. And Jason, I think you can talk about how cool that’s gonna be.

Jason (2:12:14) Well, yeah, I mean, I just I remember mentioning that to you at one point in one of our daily phone calls. And I was like, look, I’m just thinking about it. mean, because I think we were you were working, doing a lot of work in the computer science course. And I’m like, you know, I think what we would like to do is if we could create like. I don’t know we go to an algebra one, maybe we start with like algebra to intrigue or something like algebra or pre calculus or something, algebra to intrigue with coding.

Right? So you would have at least a basic understanding of variables and functions and operations and stuff. So something to build on. And having completed Algebra 1 or Integrated Math 1, whatever. And then we get to Integrated Math 2 or Algebra 2 and Trig, whatever, and we have like a with coding option. And so we have, you know, whatever, a dozen.

coding projects or more, maybe smaller coding projects, we really integrate it. So it’s not, but I think that would be super cool. think a lot of kids would really enjoy it. And some kids wouldn’t, they’d be like, I don’t care about coding, it’s just frustrating, whatever. But for the kids that do care, would be, it would probably make something that would otherwise not be very exciting for them into something that they love. And it would give them a deeper and more intuitive feel for some of it. I it was for me, I remember talking to a friend of mine who,

Years ago, he’s telling me about how when he took chemistry in high school, he wrote some coding to like do the chemical equations and stuff. And then he just like aced everything because he just, he did all his chemistry homework through coding, writing programs to solve the problems. And then he really became awesome at chemistry, which you would, because you have to think really hard about, okay, well, there’s these ionic bonds, there’s covalent bonds and numbers, and you’ve got to write variables. You’re really thinking hard about all this stuff. That’d be cool.

I think that’d be cool. I would love like pre-calculus or calculus with coding.

Justin (2:14:18) Yeah, totally. Yeah. Well, that’s like one of the things about precalculus and calculus is like, that’s, that’s the point where you’re getting into math that you can actually apply to really cool stuff. It’s like there’s kind of limited application of like linear equations because it’s so, so easy to do by hand or why do you need to code something to do that for? But when you get into precalc and calculus, start dealing with complexity that like can actually, if you code it up, you can do cool stuff. yeah.

Why not show that off? yeah.

Jason (2:14:52) I would, I would, I that that’s something that I’ve been, we’ve talked about and I’ve had in my background that I absolutely want to do. Cause you know, a lot of times we, we talk about math Academy. It’s like, we’re trying to build what we wish was around when we were in high school or college or younger. You know, we would say, God, I wish I had this. Like this is what we’re trying to build. Would I have love algebra too with coding? Yes. Well, guess then we’re going to build it. That would be awesome. No, not for everybody. And that’s fine.

You know, but that’s thing, what we’re trying to do is we’re trying to make a math learning platform that is, that can be very effective for a wide range of students, whether you’re someone, not a quote unquote math person, you struggle with math and you want to spend as little time on math as possible just to get the math done. So you can go do what you really want to do all the way up to the, the math team, super genius who wants to, you know, be four or five, six years ahead in math.

I think we can service all those people and by not only building a system that’s adaptable, but also configurable, right? So system adapts to students strengths and weaknesses, to the rate of learning, et cetera, et cetera. But it also is configurable in terms of students who want a more advanced course, or they want more competition league stuff, they want competition problems, or they want coding integrated so that we can really

⁓ kind of fine tune a course to be, to fit a student. So it’s like, if you hire again, if you hired in a really amazing tutor, like, I got three kids. got one kid who’s a super nerd loves coding loves the science wants to do all this stuff. have one who’s a good student, but isn’t like super is gift is, is the honors class, but isn’t gifted or something. And is not what likes to get a’s, but isn’t

Justin (2:16:31) in your life.

Jason (2:16:49) beyond that and then I got one who’s on an IEP and has dyscalculia. By the way, those are my three kids. So I got the whole spectrum. So it’s like I got the kid who struggled with fractions and struggled with time tables. got the whole thing. So whatever kind of kid you got, I’m with you. I know the pain ⁓ of all three. And so if you hired a really good tutor and you said, I’m going to how much you work for each three, the tutor would adapt. I’d go, okay, well.

with Izzy, we’re gonna do this. And I got some stuff that I think I can get her excited about, and we’re gonna work on her math facts, and we’re gonna work on her confidence, and we’re gonna do this. And then I got, you know, Aria, and we’re gonna get her, Colby, gonna have, we’re gonna do, MIT, you know, right, so the whole thing. So that’s what we’re trying to do, is so that we can have, and we’re not there yet, but that’s our goal.

And I think having coding courses, coding built into high school and even university courses would, you know, like linear algebra with coding. You could do that. Yeah.

Justin (2:17:57) Yeah, yeah, totally. And it’s one of the things that, like, if you know math, but you don’t know coding, you’re, missing out on a very easy opportunity to expand your surface area, to do cool things. And the sooner you can kind of expose a student to real coding, not, just like blocks, dragging blocks.

Jason (2:18:20) Or even

just putting up a webpage, like actually, like, really?

Justin (2:18:23) Actually, coding logic that does something, leveraging math, both using both math and coding as tools ⁓ to do cool stuff.

Jason (2:18:33) We always

say it’s like a superpower. Together, that’s a real superpower. Being able to write code and really understand code is by itself extremely useful. Being really good at math opens all kinds of academic ⁓ doors and is really important in the academic sphere. When you combine them together, it’s a superpower.

Justin (2:18:54) All right, ⁓ go through a couple quick ones. All right, so here’s another good one. Are you going to create a full online bachelor’s in math and or in computer science? What’s your vision for getting full college credit for every math course?

Jason (2:18:59) Sure.

Hell yeah. But yes, I mean, we’ve always thought that, right? Like, why not? I mean, it’s like, we’re getting close. By the end of this year, we should have the additional courses in addition to what we already have, differential equations, abstract algebra, real analysis, complex variables, I’m sure, as well as machine learning one.

computer science, we’ll have machine learning too, done by then. mean, even there, the computer science, the math thing, if you had all that stuff that we would have, that you could, that’s, and you know, along with different, know, yeah, getting the stuff we already have, that’s basically, that’s an undergraduate degree right there. I mean, there’s a few like elective courses, points at topology, functional analysis, complex analysis, you know, whatever. There’s a whole panoply of more advanced electives that,

all over the place that in a really good math department you could select from, smaller math departments you wouldn’t have that many options. ⁓ But we could, I think we’ll basically be there by the end of this academic year. By the end of 2026, I think we’ll, yeah, we’ll be there. Now the question is how about getting credit for it? Yeah.

There are.

accreditation ⁓ pathways to doing something like this. We’ve looked into a little bit. ⁓ You know, we looked at about a year, year and a half ago. It’s very expensive. It’s not a trivial thing to do. ⁓ But I would like I would I would I would love to do that. I that’d be pretty cool. Right. I mean, I think what we’d have one thing we’d have to do is

We’d have to make a, ⁓ we’d have to set up some sort of partnership with some of these testing centers where you take the midterms and final exams for all your courses at like, I don’t know, Pearson testing center or something like that. You go in there and you show up with your driver’s license and you check in and they take your phone and all stuff and everything’s camera. You go in, so there’s no cheating. You’d probably have to do that. there, and I mean, I I’ve done that. I’ve taken tests like that.

know, Pearson and I think there’s things like that all replace. ⁓ But I think that would be.

That would be how you do it to really make sure that people can say, you can’t cheat it. Is that not online? I know. What do you think?

Justin (2:21:57) Yeah. Yeah. We, I think we talked about this about, about a year ago, the idea of the testing center stuff. And, um, yeah, I know, um, lots of people would be particularly excited to, take these courses if, if they know they’re going to get credit for them. Now I know like one thing we talk about is like the benefits of pre-learning the material before you go into a normal college course. Cause, uh, you’re just, it’s a roll of the dice, whether you’re going to get a decent instructor or not. And, um,

and so there’s an argument there for like, well, like you should really take the course on our system anyway. ⁓ but, ⁓ but I, I know, I know there’s, there’s quite a few people who, who I’ve heard of from, who are, who are, who’ve gotten so fed up with, with their, university math courses where they’re like, do I really have, like, I pre-learned the material. Now I’m like in this like crappy, like the instructors just going theorem proof, theorem proof on the board. And like, I don’t even.

there’s barely any homework that’s assigned and graded. It’s like, why am I doing this? This is all just a charade. Like, can I just like move?

Jason (2:23:06) Yeah,

it is. it’s, it’s, uh, yeah, I mean, one thing just, I mean, you’ve talked a lot about on an axis, there’s so much, um, variance in the quality of instruction between, between just individual instructors and particularly institutions. mean, you can, I mean, some institutions are just really, um, have really strong departments and they just have higher standards. But even then, even if you go to an elite school, you’re going to get some people.

who are, or might be great research mathematicians, but are just horrible pedagogues and they don’t, they just kind of go through the motions and stuff and it’s mostly on you and you know, and they’ll just, you know, assign these ridiculously hard problem sets and midterms and really it’s you and your group of, you know, study mates and that who kind of go, hey, let’s all work together on these impossible problem sets and these, and you basically teach yourselves, you know, so it’s more like a.

framework for making you teach yourself and as opposed to them providing a real scaffolded learning experience. yeah, mean, that’s one thing we tell people. like, you if you do math academy, so I’ve had some parents talk about, you know, well, you know, they might think kid did calculus in eighth or ninth or 10th grade or whatever, and they’re doing, you know, one of our university courses and in high school.

And I said, shoot. I said, you know what I would have to do? It was my kid. I’d say do multivariable calculus, do linear algebra on our system, and then take it at the local state university or community college or wherever they’re taking it. And then they will just absolutely annihilate it. You know, it’ll be good because they’ll finish it and it’ll be presented in a slightly different way. And there’ll be, you know, there might be some hard problems here and there. And then, you know, your, your kid will feel like, like a genius.

Might be a little bored, but then they’ll help the other students. But then they just kind of reinforce stuff and it feels great. And they get and they get credit for it. It’s easy. But if you send a high school student into a university setting and they don’t realize that they’re not actually going to do a lot of teaching that they’re going to ultimately have to teach themselves and they’re not part of some like study group, we’re all kind of helping each other learn them stuff. They’re sort of like supposed to learn this their own. That can often a lot of times be really a bad situation. So they don’t really.

get that it doesn’t operate like a high school because in high school it’s like you get in theory if you’re in a good high school you have homework every night you know 20 to 45 minutes of homework and your pre-calculus your calculus course you do it every night you have a quiz every week or two and a test every month and teachers like okay this is how I keep everybody on on the path and university it’s like okay there’s a problem set do it or not or don’t do it right and they don’t even like

The grader might grade it, you might get it back two or three weeks later. And I mean, hell, I learned this in college the hard way. And we all learned it’s like you get the problem set on, you know, Wednesday, it’s do the following Wednesday and guess when you start it? The following Tuesday night. It’s like, you know, like, well, let’s do it Wednesday, I’ll it Tuesday night. Like, idiot. It’s like 15 to 20 hours of work. You’re not going to do it. You start almost Thursday, get it for dinner. So you’re going to work it out for four or five hours. Like, that’s not going to work. It’s just not going to work.

And so.

And that’s hard for college freshmen to learn who aren’t already really like strong, serious students who have a lot of self-discipline and very good at self-organizing. You get that to a 15, 16 year old, especially when they’re used to being like really, really smart and I get A’s and everything. And, you know, I don’t really have to study that hard because I’m just so brilliant. Well, that doesn’t work once you get to a certain level. It doesn’t work at all. And, uh,

and you can end up for some real crash and burns in situations. Sometimes you get an 18, 19 year old, at least they’re mature enough to realize that this is not working, you’re gonna get to change strategies. Sometimes it’s hard to get a 15, 16, 17 year old to change strategies when what they’re doing isn’t working. They’ll just ride it into oblivion. But if you already know everything, if you already know it all, then it’s like, ah! All right, this is

Justin (2:27:30) So, right. So not, and so not, not only does the pre-learning minimize your risk of, that bad situation happening, but it can also, ⁓ I mean, if you, if you are blowing the class out of the water and interacting with the instructor, like that’s setting you up. mean, if you’re a high schooler taking a college class, like you get a, like amazing rec letter out of that. Maybe you, even if you’re a college student and you’re, blowing the class out of the water, like guess who is up for whatever. ⁓

opportunities that professor has in mind. ⁓

Jason (2:28:02) Fellowship

you should apply for. We got a summer program. got a, hey, we need a grader. We need a TA, whatever. They’re like, ⁓ I got a kid. I got a

Justin (2:28:10) You just

get a reputation for being like the smart kid and like it doesn’t matter if you’re being smart in real time or if you’re smart because you’ve already built up a large knowledge base. You’re just you’re a smart kid either way and you get the smart kid opportunities and that can compound into a virtuous cycle.

Jason (2:28:27) Well, for, for my, for Colby, when he, my son Colby, who’s a, he’s not like, he’s not a senior in college when, uh, last, was like last winter break or something. He brought one of his best friends back home. He was, think, interviewing for a PhD program at Caltech. And we’re having dinner together or having dinner together. And he said, yeah, I have never been in a class, a math class where Colby didn’t already know everything.

I was like, you’re welcome. You know, and he’s like, I’m like, yeah, it sure feels nice. You just come in and you’re like, yeah, that’s easy. You know, and this friend is a very bright kid, very bright physics major, whatever. And he’s like, he had to work a lot harder because everything was brand new. Where Colby, it was just kind of reminding him of most of it. Occasionally it’d be like, I remember.

Colby would come to me and he’s like, dad, look at this. And he’d show me his class that he was taking his sophomore, junior year or something. he’d be like, I already know all this stuff.

I think you probably learned like 70 % of it. I’m sure there’s about 30 % that you didn’t actually cover. And you probably are really remember 30 % of it, right? Because of the 70, it’s been a couple years, you know, so it’ll be a nice review. You get in, the first month is gonna be a really easy, then I’ll start to pick up around the first mid, after the first midterm, and you’re like, oh yeah. But you’re just gonna resuscitate that.

knowledge and then reinforce it and build it back stronger. ⁓ it’s kind of a not easy A, but you know, as long as you put in there in some effort, it should be a solid A and then you’ll feel, know, but yeah, I mean, that’s an incredible position to be in because, you know, for most of us who, for any of us who’ve been like a math or physics major, and especially if you went to a place that we had a lot of top-notch students, I mean,

And stuff is, and you’ve seen it, you’re learning stuff for the first time and they are going at a breakneck pace and they are not playing games and they don’t get retakes and there are no studies, you know, it’s just boom, here you go. And then the average score is a 27 on the midterm. It’s like, jeez, you know, it’s brutal. And then you find out that like a bunch of the kids had actually, oh yeah, I took this at the state university when I was in high school. And you’re just like, what?

What, you guys, wait, half you guys already taken this? This is bullshit. You know, it’s like, we’re in a Spanish class and you got like a bunch of the kids who actually speak Spanish at home. You’re like, why are you in Spanish one? Your Spanish is, I don’t really write it.

Justin (2:31:16) It’s like you get gaslit into thinking you’re dumb and everyone’s just like learning so much faster than you and then the glass shatters you realize they already came in.

Jason (2:31:26) Oh, you guys all got the cheat codes? Oh, great. Okay, okay, okay. Now I get it. I get it. But anyway, the cheat code really is you’re saying, what’s the cheat code? The cheat code is learn the material before you take the course. And if you really learn it on something like Math Academy, I don’t mean just like read through a book for a few days or four, but you’ve mastered it in course, then you feel like a genius. It takes the stress out. It’s a lot more fun.

Justin (2:31:30) Yeah.

Jason (2:31:56) You go to study session and you’re helping people and it’s occasionally a hard problem. Like, oh yeah, that is an interesting problem. Oh yeah, let me think about that. But everybody’s looking to you and it’s great. It’s a great feeling. I recommend it.

Justin (2:32:10) I should just to clarify for anyone listening, but the point of learning ahead of time is not to sit there bored in class. It’s so that you can actually like legitimately grapple with the hardest problems and actually extract learning out of those in an efficient way. And so you can, ⁓ you can be the go-to person for everyone needs help with the class. You’re that person. You’re getting reps on teaching this material to your friends. You’re making connections with people. You’re in the,

front running for any opportunities that the professor has in mind, whether it’s research with them, research with one of their buddies, whether it’s internship with ⁓ a company they have a relationship with a fellowship, you just you never know ⁓ what it’s it’s it’s gonna be. But if you are in that that kind of like, just like, if you’re the go to person for for the subject knowledge, you just get pulled into all these interesting ⁓

and advantageous opportunities that just compound one thing into another. Guess what? You got a great internship at a company this year. Well, your next year internship is probably going to be even cooler because now you have this experience that nobody else has. ⁓

Jason (2:33:21) It’s

the it’s the sort of snowball effect. Yeah, starts to happen. The compounding effect is what is it’s just yeah, that’s a really good point about well, the thing too, it’s also like another type of space repetition. So you took this course on the system, you learned it, you know, maybe maybe a year goes by, maybe a few months go by and then you retake it in the school. It got a little little rusty because, you know, you didn’t do it last week.

Again, it’s coming at it from a different angle, presented a different way. Notation might be a little different. You know what I mean? And by the time you get to stuff at the end of course, it really has been a while. And then it just comes out in history. That’s what I was like with Colby. was like, you you redo abstract algebra and stuff. I mean, it’s going to be so solid. You don’t just do it once. You did it twice, separated by like three years. Right. And then it’s just like you’re

not going to be like, yeah, what’s a group again? What’s a field? What’s a, no, you just like, bam, you know, which is great. Cause you actually retain the information. The whole point of this is to like learn it and remember it, not just like learn it and just put it up, it on a resume and say, did this at one point. have no idea what it is anymore. So yeah.

Justin (2:34:35) Yeah, yeah, exactly. And this, this, the compounding of the, of the advantageous opportunities really buys you a lot of time to figure out what exactly it is you, you want to do. ⁓ you, you can afford to kind of take some more exploratory time earlier in life and kind of settle into your niche. So sometimes like, sometimes people ask me like, wait, Justin, you were teaching for all these years. Like, when did you like learn?

how to code and stuff. Like you mean you just like didn’t know how to code and then like you just like were teaching with math academy and then you just like got the opportunity to like help build the system. Like no, no, no. What I have to remind people is like, no, no, that was after I had worked for several years as a data scientist during college. Why was I able to do that? Because I had learned the like full undergrad math degree. Like I pre-learned it basically coming into my first year. Yeah, open coursework.

Yeah, exactly. Exactly. So like just you, you, you get ahead, you, get time, you get opportunities. Um, and you, you have time. I was able to say, like, after I was working as a data scientist, like, Hey, this is still pretty early in life. So I, I can afford to go like, you know, I don’t, I’m not really connecting. My soul is not connecting with, with this job. I’m going to go try this other thing that I’ve interested in. And, and then, so do that for a bit. Eventually things merge together into your little niche and it just.

It buys you more time to find that because if it takes you too long to find that, then you never actually do find it because you have to pick something.

Jason (2:36:08) Something, pick a gotta pick a major, gotta pick a job, you know.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.