Watch on YouTube or Listen on Spotify

This is a Math Academy “Wrapped” for 2025, focusing on the content side of things. In summary, here’s the good:
– We released a Discrete Mathematics course.
– We added hundreds of “missing middle” topics to our SAT Math Fundamentals course to bridge the chasm between what’s in standard school curricula versus what’s tested on the SAT.
– We soft-launched a SAT Math Prep course that students automatically promote into after finishing the fundamentals course, where they see their estimated SAT score instead of a progress percentage, and they do even more SAT-specific training such as taking frequent mock SAT practice exams and doing rapid-fire problem practice to build up speed and comfort with all the slight variations in the ways that questions could be phrased on the test.
– We added tens of thousands of free response questions throughout our middle school and high school courses.
– We developed all the content including coding projects for our first machine learning course, to be released once the coding interface is ready. (If you’re waiting on that course and absolutely must start your ML journey right this moment, note that there’s a freely available 400+ page textbook that I wrote while teaching this stuff manually in the school program – it’s called “Introduction to Algorithms and Machine Learning.”)

Of course, we’re under no illusion that we need to ramp up our rate of course production and transition from a workshop to a factory. We started pursuing that goal last year, and while there has been much pain from hitting our heads on basically every ledge possible, we’ve learned a lot and have just recently, in the past couple weeks, hit an inflection point where the factory transition is finally coming together. As Alex summarized in a recent post on X: we’re working tirelessly to upgrade our course development pipeline, building new tools and processes to help us manage a higher volume of courses so we can increase output while maintaining the quality you've all come to expect. In particular, we’re using our nearly-finished Differential Equations course as a guinea pig to test-drive some of our new tools and processes. This is the year that Math Academy comes out of the basement and onto the factory floor.

0:00 - Introduction
3:57 - Added 115 “Missing Middle” topics to SAT Prep
6:06 - Integrating the SAT Missing Middle topics into other courses
9:42 - Added tens of thousands of free response questions
10:34 - Free response questions are useful because they don’t prime you
13:33 - When to use free response vs. multiple choice questions
14:54 - Too many free response questions taxes learners
16:39 - Limiting the length of free response answers
18:08 - Building infrastructure for free response questions was a beast
20:42 - SAT test prep course
22:22 - Machine Learning has been the hardest course to develop so far.
23:12 - People who know machine learning, math, and how to teach them are rare
25:06 - The Eurisko book was the best resource for developing the Machine Learning course
28:51 - Balancing repetition and computational load in Machine Learning problems
29:43 - Designing minimum viable problems for Machine Learning
33:53 - Building the infrastructure for dynamic select questions was a nightmare
36:12 - Dynamic select questions are good for proofs and university-level math
38:03 - The Differential Equations course is almost finished
40:23 - Iterating on course development to make better courses
42:00 - 2026 is the year of scaling up course production
43:03 - How to scale up the team without sacrificing course quality
44:39 - Learning the hard way about hiring too quickly
46:20 - Challenges of managing a fully remote, geographically dispersed team
48:54 - Building tools to measure company output
50:06 - Optimizing content writer performance is like optimizing student learning
52:31 - Incentivizing content creation to improve output
56:36 - Courses planned for the longer term
58:01 - You need to learn concrete computations before abstract proofs
59:32 - Why we separate university-level courses into computational vs proof-based
1:01:07 - The best textbooks for beginners are NOT the most complex
1:02:37 - Teaching proofs and computations at the same time overloads most students
1:04:16 - Intuition through repetition
1:04:49 - Wisdom is the abstract compression of lived experiences
1:07:39 - Mastering details before abstracting

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

Watch on YouTube or Listen on Spotify

The raw transcript is provided below. Please understand that there may be typos.

Justin Skycak: (00:00) Welcome to the Math Academy podcast. I’m Justin Skycak, Chief Quant and Director of Analytics at Math Academy. And I’m here with our founder, Jason Roberts, and our Director of Curriculum, Alex Smith, for a Math Academy wrapped for 2025. In this first part, we’ll focus on the content side of things. In summary, here’s the good. We released a discrete mathematics course.

We added hundreds of missing middle topics to our SAT math fundamentals course. These topics bridge the chasm between what’s in a standard school curriculum versus what even more advanced skills show up on the SAT. And along those lines, we also soft launched an SAT math prep course that students automatically promote into after finishing the fundamentals course. In the prep course, they see their estimated SAT score instead of a progress percentage.

And they do even more SAT specific training, such as taking frequent mock SAT practice exams and doing rapid fire problem practice to build up speed and comfort with all the slight variations in the ways that questions could be phrased on the test. We also added tens of thousands of free response questions throughout our middle school and high school courses. And we developed all the content, including coding projects for our first machine learning course to be released once the coding interface is ready.

And by the way, if you’re waiting on that course and you absolutely must start your machine learning journey right this moment, then just a reminder that there’s a freely available 400 page textbook that I wrote while teaching this stuff manually in our school program. The textbook is called Introduction to Algorithms and Machine Learning. Of course, we’re under no illusions that we need to ramp up our rate of course production and transition from a workshop to a factory. We started pursuing that goal last year and

While there has been much pain from hitting our heads on basically every ledge possible, we’ve learned a lot and have just recently, in the past couple of weeks, hit an inflection point where the factory transition is finally coming together. As Alex summarized in a recent post on X, we are working tirelessly to upgrade our course development pipeline, building new tools and processes to help us manage a higher volume of courses so that we can increase output while maintaining the quality that you’ve all come to expect.

And in particular, we’re using our nearly finished differential equations course as a guinea pig to test drive some of our new tools and processes. This is the year that Math Academy comes out from the basement and onto the factory floor. So let’s get into what we learned in 2025 and where we’re headed next.

So Jason, you mentioned a couple of weeks ago that maybe we do sort of like a beginning 2026 year in review and year ahead sort of thing.

Jason Roberts: (02:48) lot has happened. ⁓ We could probably go through the good things, the bad things, I don’t know, whatever. The things that were on the docket that didn’t make sense.

Justin Skycak: (02:57) Why

don’t we start with good things? Why don’t we start with ⁓ what we actually pushed out? Well, so I know one of the good things is, I mean, we talked about this previously, but just to summarize here, the extra 115 topics or so in the SAT Fundamentals course to kind of excavate the missing middle. So SAT Fundamentals course is like, that’s a legit course now. It used to, I mean, it’s been on our course list for a while now, right? But it used to be kind of underpowered, but now it’s like,

Okay, this is the legit thing. And the process that we figured out to do that, can apply to any sort of exam. So ACT, that should be a layup once we get into it, but we can see how far we can push this sort of thing into competition math and whatever. But of course all the other exams too, Lots of different countries have lots of different standardized exams, so there’s a lot of kind of low-hanging fruit just like the SAT, ACT sort of stuff.

Jason Roberts: (03:57) So the original version of the SAT fundamentals just didn’t, didn’t fully cover everything. know, mean, what it covered, was everything that was in the standard curriculum that you need to know for the, do well in the SAT, but it didn’t cover everything on the SAT, everything you need to do on the SAT. There’s a sort of gap, you know, that exists. And, you know, if you were, ⁓

If you did what we did initially, which is just, remember Alex just looked through the description of the topics covered on the course and he’s like, well, look, we got this stuff and we had to get something together really quickly. so then we said, all right, we’ve covered, this is a good first shot, but over time we realized that we were missing, missing some things. And it would probably be enough in the course to get to a certain level score, but that to get, you know, maybe to a seven, in the seven hundreds to an eight hundred, you’d

probably need to know more. And so what we ended up doing, well, it was funny. did this sort of, of, played this sort of like little scenario in my head. It’s like, imagine you hired some really expensive SA2 tutor, a tutor who is known, who had a great reputation for giving, kids, helping kids get these amazing scores. And you had this guy and you’re like, so what do you do that’s so special? And he’s like, well, I have this golden list of topics. I’ve been coaching this for years. And there’s, you know, after I get the students,

foundations up to speed, there’s this whole series of skills they need to have to really do well. And if you had that conversation, you’d imagine like, what are those topics? What are these golden topics that this guy has? He pulls up his folder and he’s like, for example, and he starts going through it.

Justin Skycak: (05:40) Yeah, stuff that’s outside of the standard curriculum, stuff that you’re not going to learn in your school class, stuff that is kind of like this privileged knowledge that these experienced test prep tutors have that really it’s basically just the families who can pay an arm and a leg for test prep get this sort of information. yeah, besides the SAT missing middle stuff, like what other content did we put out in 2025?

Alex Smith: (06:06) Yeah. So, um, yeah, so we, as you, as you mentioned, we had, had it like 120 or so missing middle topics, uh, to the SAT fundamentals course. And the plan, one of the plan, I’m leaving ahead here slightly, but one of the plans for 2026 is to integrate the vast majority, vast majority of that into the, into the, uh, the standard high school curriculum. So if a student is doing our standard curriculum,

ready for it they complete it up to a certain point, maybe say Algebra 2 or Integrate Math 2, they’re ready for the SAT. They haven’t got to do some additional SAT fundamentals course in order to kind get access to those extra stuff. They’re to be hard baked into our mainstream curriculum. A few topics have made it in there already for kind of connectivity purposes, but that’s something we’re planning to do.

Justin Skycak: (06:53) Yeah, that’s awesome. So really, so it’s like, I mean, it’s, one of those things. If you just go through the standard curriculum and you’re not doing these golden topics, these missing middle topics, then by the time the SAT comes around, you’re like, Oh gosh, I have like 150, 200 topics that are like just off of the standard curriculum. So this, this extra 115 topics that we added, that was kind of like, in addition to some topics we previously had in there, right.

Alex Smith: (07:20) Yes.

Justin Skycak: (07:20) Yeah.

So there’s a really, like, this is just the, the, the latest batch of the missing middle. There’s, more of this thing besides this. And yeah. So if you’re waiting until like, Oh, Hey, I got like, I got, I’m taking the SAT and in two months, maybe I should prep for it. And you’re like, well, you got like 200 topics to learn, dude. You, this is not a good situation. So it’s better. And especially even if you did go really crazy and you were like, okay, I’m doing like 10 topics a day for this week. Like I’m doing like hundreds of XP per day.

And like, you’re not going to have time to really consolidate it. And you’re probably, I mean, you’re just going to be doing so much that it’s really difficult. It’s like going from like, ⁓ I barely know how to cartwheel to I can do a backflip in like two months. Like that’s not going to happen. got to space that out over more time. So yeah, that’s, that’s awesome that we’ll have that in the, in the main curriculum too. Yeah. It’s a great topic, Like, like just beyond just like the SAT, like, I mean, this is not, this is not just like,

Alex Smith: (08:10) I’m really excited about

Justin Skycak: (08:19) tailored to like, there’s just this one really specific sort of thing on the test that like, doesn’t really further your learning overall, but it’s just this thing. No, it’s like, it’s kind of more, it’s really, it’s almost like challenge problems in a way, right? Like ⁓ compared to the standard curriculum.

Alex Smith: (08:37) Yeah. I mean, I suppose with some of them, it brings together prerequisite knowledge in ways that are kind of slightly unusual. you might have some topics on analyzing, know, sort of comparing two data sets or something like that. And yet it’s bringing in things like, know, sort of mini-max and things like that in kind of very subtle ways. So yeah, I think it’s going to help.

Anyone that has this knowledge in their repertoire is going to definitely benefit them going forward and studying more advanced stuff. It might even be the case that some of it might end up being prerequisites of some other topics that we have further on down the line. There’s going to be a bit of a connectivity baking when we have this. Most of it are leaf nodes, but there’ll be some that actually…

That’s a really good prerequisite for this topic that already exists, you know, that we didn’t quite think of. ⁓

Justin Skycak: (09:35) That’s awesome. What other kind of content? I know like we had a lot more free response questions in the system, right?

Alex Smith: (09:42) so this is in the past year. we’ve, we’ve, we’ve added like tens of thousands of free response questions, ⁓ over the past year, just cause just continually kind of like chugging, sort of chugging away, adding free response. ⁓ and we’re almost, we’re at the point now where almost all of the, ⁓ so all the middle school, ⁓ courses have them, like most topics have them and almost all the high school, ⁓ topic at high school courses too. So I think we’ve got up to about sort of like.

sort of free courses away through calculus one. So, or AP Calc, so halfway through sort of AP Calc. Once we get to the end of AP Calc, that’s the whole high school curriculum with lots and lots of free response, as well as the middle school curriculum, we just had them for a while. yes, that’s been a big thing. ⁓

Justin Skycak: (10:27) That is awesome.

Jason Roberts: (10:28) You know, it’s funny, we should talk just a little bit about that because it’s interesting, it’s like free response. The reason we do free response is because you’re not primed with potential answers, right? Sometimes I get asked a question and you see some potential choices. You’re like, right, it’s this kind of an answer, but when it’s just there’s nothing to prime you there, all you have is an empty box. like, ⁓ is this a number or an expression I’m supposed to enter? You have to think a little more about it.

The side effect, of course, is that you have to type more in. So if you do it all too much free response, it can get a little arduous.

Justin Skycak: (11:04) So just a concrete example of this, example, say the question is, find the area of the circle. If you see pi in a lot of the answer choices, that kind of tips you off as like, ⁓ right. Like the result has pi, pi r squared. Like it kind of helps prime you into it. But if it’s just a free response box, you’re just like, well, like you have to actually remember that this has something to do with pi. You have to just pull that from scratch out of your head. There’s nothing, there’s no priming.

Alex Smith: (11:31) Yeah. one thing as well, especially it’s something, I mean, it’s something that we sort of, I guess, deal with when we come to like the in-task coaching, but certainly younger children, sort of studying like the fourth grade and fifth grade, they’ll result to like non-optimal sort of strategies to, to learn the content. So for example, you know, kind of like, sort of like…

Trial and error, know, they might, if it’s like, what’s the solution to this equation, for example, it’s like a really simple linear equation, you know, if it’s a free response, it’s very easy to take every single answer, just plug it in. it’s that one that works. It’s that one. That’s the correct answer. But of course, that’s not the strategy you want them to be doing. You want them to be solving the equation and getting the correct answer that way. Now, more disciplined students would do that anyway, but we want to discourage that kind of behavior as much as possible.

And there were some, now obviously when Math Academy first started, free response was all we had, right? So we kind of had to sort of make the best of it. Multiple choice was all we had and multiple choice was all we had. So we had to make the best of the tools we had. But one of the things I want to do is kind of go back to some of the early grade topics, our fourth grade, fifth grade topics where we’ve used multiple choice and asked ourselves, it…

Jason Roberts: (12:25) Now, of multiple choices, no way.

Alex Smith: (12:45) And then they’ll have free response questions too, but ask us, should we be removed multiple choice entirely from those topics? Because for example, if you’re multiplying two numbers together, you’re using like the standard algorithm. You know, don’t need to see multiple choice options. need to be, you know, it’s quite easy to enter four numbers and that’s what we should be testing. Should we know priming, know guessing, by sort of looking at the unit or whatever.

So yeah, so I think that we’re doing a first pass of the free response stuff and then we’re to kind of circle back and then do like a really hard analysis of, where is multiple choice a really bad thing to do? We’ve got these other options now. We’ve got free response. We’ve got like static select and dynamic select questions, like drop downs. At what point, what knowledge point should we be removing those multiple choice questions from where?

Justin Skycak: (13:33) Yeah. One thing that I think we should clarify to listeners at this point is that, so free response can be really helpful for a lot of these topics, but not all topics need free response. not a multiple choice. We still have multiple choice questions in a lot of topics because they are good to use quite often. Like for example, say you go through a really long computation.

Let’s say it’s like an integral or something and the numbers are all like, they don’t really prime you to what the result is. And there’s no way to like, plug in or to guess and check based on the answer choices. Then it’s like, well, there’s, there’s no, you don’t need to necessarily have a free response question there. And in fact, a free response question sometimes like can, it can be less forgiving to small errors. Like, you know, if you, if you

If you make a small error while you’re, you’re integrating something like you kind of, you know what you’re doing, but you just like you, you added two numbers or you dropped a negative sign or something. And then your answer doesn’t come out to one of the choices that kind of, that kind of tells you in the moment, like, okay, like, Hey, check your work. Like if you really know this, you’ll be able to figure out if you, if you don’t, then, then you won’t, but it’s, it’s like, we’re

we’re not going to penalize you too much for a little mistake. So the multiple choice questions have that kind of nice feature.

Jason Roberts: (14:54) The other thing too is that free response, depending on how involved the expression is that you’re entering, can be sort of a tax on you. I it’s exhausting to do a lot of that, right? So if they had to that all the time, and especially it was like longer algebraic expressions, it’s like, ugh, you know what I mean? If you do it all the time, you’re creating this sort of effortful tax.

And it adds up where he’s like we want you to keep going right? So we want to kind of do multiple choice where we can get away with it or and then and do free response where we can get away with it. Like let’s use each to the maximum advantage like we need to sometimes primary you need to do the whole thing and enter it other times. Like you said, you know it’s a little the multiple choice resilient is a little more robust against silly arithmetic errors, but it just it’s you can go a little faster. We’re trying to get as much out of the student.

as we possibly can in amount of time they’re going put in the system.

Justin Skycak: (15:54) It’s like most things, often like a combined strategy. You got two parts of the strategy that are good at different things. And when you combine them into like, OK, let’s have a mixture, have a variety of questions. Generally, a variety of question types is good. Variety of training tasks is good. And it’s kind of like we do with reviews versus quizzes. Not every review problem is under time conditions like closed book, like super.

pressures on. It’s like, well, if we made every single review problem in a quiz setting, like people would burn out pretty quick. But ⁓ it’s good to have quizzes. It’s good to have enough of quizzes to be kind of like a speed bump when you need them. But you don’t need every single review question to be like in a quiz setting.

Alex Smith: (16:39) Yeah. I’ve actually also given some instructions to the team about, you know, limiting, the instructions to my content team about limiting the amount of expressions we, you know, or how big the expressions and things like that should be in free response. If it’s like a four or five term polynomial, know, a fifth degree polynomial or something like that. It’s like, that’s a real, that’s gonna be real pain for students to enter, especially if it’s like one after another, after another. like, no, three terms, you know, if it’s polynomial, three terms only, preferably, three terms max.

Um, things like even, even numbers, like if it’s like, you know, if it’s like one, if it’s like one over root two, like that’s totally fine. But if it’s like 51 over the square root of 353 PI, something like that is like that’s unnerving for the student. They’re sort of entering a number like that. And it’s like, is this right? Is it right? It doesn’t feel right. It doesn’t feel too nice. I can’t say that, you know, nice, nice numbers only, you know? Um, so a week or so ago, I kind of, cause we’ve just got to calculus and calculus is

the point in the curriculum where you start encountering expressions like that, like derivatives using the chain rule and product rule and stuff. It’s like, look, don’t go crazy with the free response for these very complicated expressions because it’s unnerving for the student. It’s a pain in the backside to type. Just, you know, apply it where it’s necessary. And you can’t really back engineer a derivative if it’s like free response, if it’s multiple choice, for example. So yeah, it’s like, again, it’s that intelligent use, using the best tool for the job at any given moment.

Jason Roberts: (18:08) There’s a bear building that technology though.

Justin Skycak: (18:11) Yeah, remember when we first started out with the free response parts, we were like, okay, this isn’t going to be too bad. We just like read the LaTeX and then evaluate the expression on a few sample points and then like see if that, if it comes out to the right number. I mean, that’s still, I mean, that’s kind of the general principle that we use, but it’s just very, there’s lots and lots of edge.

Jason Roberts: (18:35) some more complex and that to handle more sophisticated functions and stuff which can really transform or be more affected by your domain and domain and range issues and sort of like sensitivity sort of error sensitivities to things that can really mess things up. I mean, and then there was just, you know, you kind of have this, we have this sort of algebraic equation parser, but there was so much pre-processing has to go in to handle the huge range of what

expressions can be entered, you know, because it’s not just algebra, algebraic terms, it’s all the trig and exponential and everything else and every combination of those and roots and exponents and I mean, it’s crazy, right? But it was funny, mean, Justin, you kind of took it over time, you just would, because there’d be like some kind of complaint from the content team, it like, McLaurin expression isn’t evaluating what is going on, you know?

you would get in there and have to write all this pre-processing to say, okay, well, what this really means, we have to transform it into more of an algebraic form so that the system can handle it.

Justin Skycak: (19:42) Yeah, it really turned into like an algorithmic sort of quant like task. Cause it’s the long tail of weird edge cases and weird behavior.

Jason Roberts: (19:54) But it’s gotten fewer and fewer over time. I it’s been a long time. It’s been a while since we’ve had something, but you’re right. was just kind of like, like it was fine. And then like, you know, and then like for like a month, I’m like, this thing is pretty solid. And then it was like, Oh, there’s something. And then, you know, it would kind of go longer, but yeah, that’s great. Anyway, I’m glad we have it I’m glad Alex that the team has made so much progress and, and, um, inserting these questions in this, in the content. Cause I think it really increases the quality.

Justin Skycak: (20:24) So I want to circle back around to more question types later, like the dynamic select stuff and also the upcoming coding questions and stuff. why don’t we, so Alex, did you mention that there was more ⁓ content stuff that you could talk, what other content stuff have we added to the system in the past?

Alex Smith: (20:42) Yeah. So, ⁓ so the, we’ve got the, so as well as the SAT fundamentals course, we’ve also got the, SAT test prep course, which is public launches imminent is there, it’s ready to go. We just need to launch it. So that’s going to be coming very, very soon.

Justin Skycak: (20:58) What’s left on that? Well, I think we got all the topics, right? We got all the topics, and we got the estimated score. We got mock exams. So I think there’s just a, what is it, just some kind of communication to the student on what the course is when they enter into it. Because we added it to the course sequence that has the SAT fundamentals. So if you finish your SAT fundamentals course, you will promote into SAT prep.

Jason Roberts: (21:29) Yeah, I think all we need is the course description. Alex is going to write up the course description, but that’s a couple hours of work or whatever at most. And, you know, the outcomes, the description, overview outcomes stuff. And then, and then I think we just got to put some communication. I can update the website. I mean, it’s basically ready to go. It’s been ready for a while. We should, we just been so preoccupied with all these like, it seems like we always have like one emergency after another of something we have to do that just.

Alex Smith: (21:49) It’s ready.

Jason Roberts: (22:00) So, yeah, we just need to take a half a day and just do those last few things and it’s ready to go.

Justin Skycak: (22:07) Yeah, let’s do that. Easy win. Yeah.

Jason Roberts: (22:09) Yeah,

easy win.

Alex Smith: (22:11) ⁓ so was it released, so in terms of upper core, upper division course, we released a discrete math. I can’t remember when it was back in March, something like that, February, March. ⁓ and the big one is, although it’s not, it’s not available for public consumption yet, but the machine learning one course content wise is pretty much ready to go. And that’s been, that’s been a huge undertaking. ⁓

Probably, I mean, almost certainly the most challenging course to date in terms of how to, how to deliver that kind of content. And that’s without the coding projects, which I haven’t actually been that much involved in.

Justin Skycak: (22:46) speak

of it to why that was so hard. mean, yeah, that was so, that was, yeah, right. The most challenging course we’ve ever done by far. why, like to listeners who may not have tried to develop a machine learning course themselves and been in the weeds and gotten like punched left and right by the universe on this thing. Like, why is this such a formidable opponent?

Alex Smith: (23:09) Well, this is a really interesting question. I think one thing is it’s quite difficult to find people who’ve got both the machine learning sort of like knowledge, but at the level we need, which is the math chops essentially. Like those who really understand the mathematics behind this algorithm at a very, very fundamental granular level. It’s hard to find people like that. Machine learning engineers do seem to be kind of too a penny.

But people that really understand the math, they’re much.

Jason Roberts: (23:40) What’s that phrase used? That’s an English phrase. A two a penny? So they’re dime a dozen, as we say. Yes. They’re dime a dozen. So right. So you can find a lot of people who can have experience with numpy or whatever, pie torch. Not numpy, but pie torch and tensor flow and can talk about these different, at a high level, how these algorithms work and what circumstances tend to work. they’re good. They’re practitioners, right?

Alex Smith: (23:44) To a penny, yes.

Yes, yes.

Jason Roberts: (24:09) A practitioner, an engineer needs to have certain types of skills, other skills, really understanding the mathematics at deep level isn’t always necessary. I mean, depending on what they’re kind of doing. And so if they’re not using that stuff, and they didn’t take up to, you know, get a graduate degree in this stuff, or even if they did and they haven’t used it in a while, it’s just atrophied, right? So you, you talked to a lot of people who, but they just didn’t quite have the math.

Alex Smith: (24:32) Yeah,

I mean, there were, were, for, by all accounts, like, you know, decent machine learning engineers in their own right, done projects for other people who were very happy with them. But it’s just, it’s just like, I, the, the, the, the, the understanding of the math, which as you say, they don’t necessarily use day to day, or they might’ve studied it years ago and forgot or whatever. And then there’s, and then there’s like the pedagogical side of it. It’s like, how do you, okay, you do understand that the math’s on a very fundamental level, but how do you actually present this to.

the how do you present this in a kind of ⁓ in our kind of setting? Now, we’re really lucky because I was adjusting wrote a book about I’ve got it here. Look at this.

Justin Skycak: (25:14) Yeah, you got it. There is. Introduction to algorithms and machine learning. This was the book for the Urisco sequence, the super advanced computer science sequence that Jason and I were running within the in-school program.

Jason Roberts: (25:29) This is at the high school. This is what for high school kids. So the thing about what’s great about that book though is Justin was just making this stuff up in the fly. He and I would brainstorm and scheme like, oh, maybe we should teach this. We should teach A star. We should teach Kalman filters or we should do whatever. We should teach genetic algorithms. And he’s like, all right, let me think about it. And he would come up with stuff. And he would write up some stuff and pull stuff together and come up with some lessons or some exercises for the kids to do.

And then, but he’d battle test that with kids. if you can’t, you know, it’s a kind of a high bar because you’re teaching stuff that is advanced undergraduate and sometimes even graduate level concepts to like 15, 16 year old kids.

Justin Skycak: (26:11) Yeah, we went all the way up to reimplementing actual research papers and artificial intelligence, these 90s style artificial intelligence, like the Blondie 24 program or training neural nets using evolution to play games. Like, I mean, started off tic-tac-toe, but then checkers and more advanced stuff.

Jason Roberts: (26:31) Yeah, it was crazy. I mean, it was such cool stuff. But when you’re asking Justin to sort of make that happen, I mean, that’s a tall order. So he had to really think hard about how to do this stuff. And there was a lot of trial and error. Like some stuff worked better on the first try. Some it took a few times because you went through three different classes, three different years. And by the third one, you had…

kind of fine-tuned the process, the description of the introduction of the content, the exercises, but then you actually end up, I think you spent a good part of one summer just writing up all of your notes and putting them to a bunch of PDFs. Right, so by the third one, the third year, kind of like here, here’s the PDF, this is the whole thing.

Justin Skycak: (27:12) Yeah. Yeah, exactly. Yeah. Cause I was tired of, well, I was like, okay, I know exactly what I’m going to say and what problems I’m going to give you guys. Like this is kind of nailed down. So why am I just going to do this manually again? That sounds like a waste. And then also like the, since I was relocating and the program was going to end, like the natural question as well, how can students like kind of like, ⁓ learn the, the, the class material if I’m not there to teach it? Well, like at least, at least I can say like, okay, here’s this book for you guys. If you’re

You wanna learn it? Here’s the material.

Jason Roberts: (27:45) Having that as a foundation for the machine learning course, think, Alex, I didn’t hear that. I how helpful was that compared to just whatever else was available?

Alex Smith: (27:53) It was super helpful. mean, I’ve got all the big machine learning books, you know, but this is the one I use the most to actually help with this course. At least in terms of the philosophy of how this stuff is delivered. Some of it actually reminds me a little bit of teaching like numerical methods. You know, it’s like you’re presenting an algorithm and you kind of work through the algorithm kind of step by step. know, if it’s like Newton’s method or something like Newton’s root finding algorithm, you know, you write down the expression for, you know, the iterative procedure.

and find one, do one successful iteration, do two successful iterations. Now do the whole thing until you get it to converge. And it’s like, great. Now you know Newton’s method. know, so it was, was, was the book plus a little bit of my own sort of experience in teaching like numerical methods and stuff. It’s like, okay, this is, okay, I get how this is going to work. I mean, it’s, it’s, there were, there were a lot of parallels. So the task really was to take this, this idea and just put it into our, our system.

Justin Skycak: (28:51) Sounds like kind of a delicate balance of like, okay, students have to do the reps, right? They have to work things out by hand. They can’t just like, here’s a formula for this method or that method. Here’s the neural net back propagation formula. Now just like, now transcribe it into your Python code. Great, now you know back, no, you just transcribed it. You don’t actually know it. You actually have to work through with some numbers, like see how everything flows, get a sense of.

you know, just what things look like intermediate steps. And, but at the same time, like there’s, there’s a way to do this, but then make it like hell for the student because every problem takes like 20, 30 minutes, cause you’re doing so much. So you need to avoid that also. So it’s very, how do you, how do you get students to do their reps, but at the same time, not just crush them with like the computation burden, right?

Alex Smith: (29:43) Exactly. That is the hardest bit. So you’ve got to think about the sort of almost like the smallest vibe, the minimum viable problem that they can do every single time. So it’d be like, it’s a neural nets, for example, you might, if you’re teaching like back prop, it’s going to be like the smallest neural net you can get away with, with specially picked numbers so that the numbers kind of work out right. you know, ⁓

Justin Skycak: (30:04) The

number picking is kind of interesting too, because sometimes that’s counterintuitive, right? Like you might think like, well, I give you inputs of 1.0, 2.0 and zero. And then you run that through and suddenly by step two, you’ve got these nasty decimals because everything’s been like log transformed or whatever. So sometimes you have to choose the numbers as like in order to get nice numbers, you say like, okay, your inputs are natural log of two, natural log of three and one. And then those look so bad like initially, but

By step two, it’s like, well, they’ve converted into very nice whole numbers that you can just kind of take with you.

Alex Smith: (30:40) Yeah. I mean, for me, my biggest, mean, I’m not a machine learning expert. I learned a lot from doing this course, but probably my biggest battle was, mean, eventually, like, I mean, after we spent some time, we managed to pull together a team that was capable of delivering this stuff. It took a few iterations, but we managed to. And my biggest battle at that point was just making sure that exactly this type of thing didn’t happen. Like we were creating problems that…

It was just obvious the student was going to be struggling with them like 20 minutes, 30 minutes on one problem. Like this is, this is not good enough. So that was, that’s why I spent most of my time doing just simplifying the problems, modifying the inputs, ⁓ just changing the scope in such a way that it’s, something that you’re getting genuine reps on the algorithm, but not in a way that’s going to kind of crush you.

Justin Skycak: (31:26) Yeah. And that’s like basically nowhere else online, right? Like you look up these algorithms online, like you try to look for an example of like, can you just give me an example of something simple, like just a decision tree, just how do you fit a decision tree? Just give me a concrete example. And like basically nobody has a good concrete example online. It’s either something like that is so overly simplified that it becomes trivial and you don’t actually get to work through the algorithm or it’s just, well.

Or it’s just somebody gives you like a ⁓ notebook tutorial and then you just copy the code over. And if you want, you can inspect the outputs and stuff, but it’s so like awful and messy. There’s nothing, it’s so hard to find middle ground stuff.

Alex Smith: (32:10) Yeah. This is the advantage of having like an online system. You mentioned coming back to some of the other question formats. I’m going to sort of jump ahead slightly. Like we’ve got like one question topic is dynamic select, which is kind of like, it’s almost like a multi-part problem, but in a regular topic. you do, for example, if it’s like, know, using like fitting some sort of decision tree and you have to compute the Gini

indexes like at various, as you kind of progress down the tree. ⁓ So yeah, so you start off with like the first part of the calculation. And then if you get that right, it unfolds to the next part of the calculation. And then if they get that right, you unfold to the next part of the calculation. Do that right, you’re done. And the wonderful thing about that is it’s a wonderful pedagogical tool because it really allows you to kind of almost like force students down the path you want them to, the path you want them to go.

Justin Skycak: (33:09) Yeah, like work out. It’s like show you work out the problem in the correct way. And by the way, this is the only way that we’re going to accept your input. have to do it.

Alex Smith: (33:18) Exactly.

Yeah. I mean, I guess it’s tricky with that, in some other topics I was doing recently, sometimes the answer is like quite obvious. know, for the, you know, the Laplace, not machine learning, but this is differential equation, like the Laplace transform of E to the KT. The answer is always the same. It’s like, well, how do you test that students know that? It’s like, well, what you can do is you can force them through the, the steps to derive that using these kinds of dynamic select questions.

And through that, they can get lots of reps on exactly that process. So, I mean, you can.

Justin Skycak: (33:53) Even building the interface for the dynamic selects, was kind of a thing, right, Jason? mean, it’s, but like.

Jason Roberts: (34:00) That was,

that was, that was one of those super hacky kind of things where you just got to get in and be like, all right, like I’m about to get real nasty here, you know, cause, cause you would have selects that could be just embedded within like the flow of the text. Right. So select itself was like a dynamically, it wasn’t like a ⁓ standard element. was a, like a custom div with, you know, what looks like a kind of a drop list, but, but in there.

Because you’d have to be able to render in there the options, actual LaTeX, like equations and stuff. I mean, not always. Sometimes it’s text, sometimes it’s LaTeX. It has to size itself correctly, you know, and position and frame itself correctly. But then sometimes you would have like a block of math, of LaTeX, and then embedded in there would be the select. So I have to get in…

to the math, LaTeX pars this mountain of elements in all their custom tags and just crap and be like, ⁓ what am I gonna insert in here? Because they’re rendering too.

Justin Skycak: (35:11) Yeah. There’s might be like a system of equations where you’re like entering a couple of terms in one of the equations. And this is not just like on its own line. It’s got the big, like, you know, the big brace to indicate it’s a system or maybe it’s like matrix entries or something.

Jason Roberts: (35:26) You have to fit like what how big is the thing externally? the thing might so if I’m sizing stuff inside and then it has braces if I if I assume it’s this big but the braces are that big or the braces are this big and things that looks stupid, right? So it was just it was the nightmare. I I mean, I also

Justin Skycak: (35:46) Late tech, when it renders this stuff, sometimes it intelligently changes the sizes of things. So it’s like, this is not just the hard rule of like, anytime you have a matrix bracket with two entries, make it this big. No, it’s like, it’s very, very complicated.

Jason Roberts: (36:01) I mean, it’s not the most beautiful code in the world by a margin. It’s like, it works. I got it working and it works. I mean, that works great. Right. I mean, it just, right.

Alex Smith: (36:12) It’s wonderful for these kind of, as I mentioned, like this, especially like the sort of upper division stuff where you really do need to kind of like get, get students to kind of demonstrate their knowledge in, in multiple steps. Like the decision review stuff we talk about. can’t wait to see what students make of that. think it’s, I think it’s fantastic. Um, like I said, like with this, past transform stuff, I think we’ve got quite a bit of that in the discrete math course. That was probably, oh no, methods of proof is probably the first course where we, really made use of that, but we really went to town.

with the discrete math course, like you’re running Dijkstra’s algorithm, like by hand, know, Kruskal’s algorithm, all this classed up with graphics and all this, it’s just, super cool. love it.

Justin Skycak: (36:50) I think discrete math was the first course that had to do with graphs, like nodes and edges kind of graphs, right? And those can be a pain to, you know, talking about graph algorithms and stuff, without these dynamic select questions, just, there’s just not really a good way to, to present.

Alex Smith: (37:06) Yeah.

Just going back to, mean, it’s funny how these things evolve. It’s like, mean, when it comes to the machine learning course, the book was a huge, huge help, but it’s also like my own sort of personal experience. Like, so obviously I studied in England, I did A levels and stuff like graph algorithms. I didn’t wait to university to do that. So I didn’t, I knew what Kruskal’s algorithm was before I went to study as an undergraduate.

And when you’re, so I did it at A level, is the, that’s the qualification you do between 16 and 18 here. And depending on your grades, depends on what university you can, you can get admitted into. So things like Kruskal’s algorithm, Prim’s algorithm, all that stuff, doing it by hand, you know, right on very, very small graphs. I’ve kind of done all that. So it’s just kind of taking that experience and putting it in the system, you know? So it’s just, it’s just an interesting combination of events that led to what we have, ⁓ in, in these, some of these upper division courses.

Justin Skycak: (37:59) Absolutely. Any other content that you want to talk about?

Alex Smith: (38:03) So I promise everybody differential equations course is extremely close. Yeah. Another couple of weeks and I think we’ll be ready with that. And we’ve also got an updated version of mathematics and machine learning course. Again, it’s been delayed, it’s pretty, it’s good to go almost.

Jason Roberts: (38:21) Well, you said that one thing that’s always kind of exciting is every course you create is like the best one we’ve created because we have so much accumulated expertise and improve pedagogical tools or just approaches, right? So like this differential equations course, from the way you’ve talked about it, like it’s going to be lights out.

Alex Smith: (38:40) Yeah, I’m really excited about the differential equations course. I think it’s going to be a great course. There’s some brilliant problem solving type activities in there. Just applying these models to different situations. And like I say, just things which are quite hard to teach and also can be quite dry subjects. Like I mentioned, Laplace transforms, Fourier transforms, all these kinds of things that we’re going to make use of those dynamic selects and really kind of…

get students familiar with how these things are constructed and what their properties are and how those properties make, can prove more properties. I’m super excited about this course.

Jason Roberts: (39:16) Yeah, I think this is one we’ve wanted for a while. mean, what’s interesting is that we had a lot of material because you had created some, we had sort of a proto differential equations of course for the high school program, the math academy high school program at Pasadena High School, you know, because they would teach freshman year, they would do linear algebra, multivariable calculus, and sophomore year was differential equations and abstract algebra. But those courses weren’t always 100 % complete.

because we had to get stuff out fast. I mean, we’re just laying down track right before those classes are like not right for the beginning of the year, but like week before they were in a tour, two weeks for it. I’m like, we gotta go, you know, and you were just like, right.

Justin Skycak: (39:56) this kind of runway tool that would kind of list like, okay, who is in danger of running out of topics and be like, ⁓ crap, like Alex, we got like one week before half the class is like done. Like they’re just going to be sitting there with nothing to do. Like, what do we do? I guess we need more topics, Alex, more topics now. Kids are going to flat line. They need abstract algebra. They need differential equations stat. So yeah, you would just kind of be like, let’s go. Let’s turn out what we can. ⁓

Alex Smith: (40:22) And

it was kind of like showing out what we can. And lot of it was actually surprisingly good. mean, this was years ago when before we had any of these kind of fancy like free response and dynamic select and static select. we know a lot, we knew a lot less than what we know now about how to structure these things. But a lot of it ended up sort of like sort of looking back on it. I actually, that’s pretty good. I was so, we just, we just use that maybe a couple of tweaks here, but some of it, like I mentioned, I keep going back to this Laplace transform stuff. It was just, it was just so easy to game.

⁓ you know, just pattern match. was like, I mean, this is, this might’ve been okay. Like five years ago when we’re trying to sort of just, you know, just get anything, anything we’ll do at this point, the training is coming, you know, just laying down the track. But obviously for a commercial course, you can’t have stuff like that. so yeah, so there was a lot of, a lot of the time was spent kind of undoing some of the, some of the things that we did for, for, for exactly that purpose. mean, it was great. I’m not saying we shouldn’t have done it because it gave us information about what works and what doesn’t work.

But certainly some of it was like, yeah, this is not commercial, Greg. We need to go back and fix this.

Jason Roberts: (41:25) Yeah, yeah. Well, that’s been the case, I think, for a lot of the courses that they benefited from that. I remember when we came to the linear algebra course, and the discrete math, and the probability statistics, everyone was like, this is the best course we’ve ever created. This is incredible. I mean, that’s been generally your feeling about every one that’s come out over the past couple of years. They just keep getting better and better, which is…

Justin Skycak: (41:46) Yeah, it feels almost like Formula One racing. Like we’re just going to make the coolest, like make it even cooler, make it even cooler. And ⁓ so that’s something that we’ve been doing a pretty good job at and just like upgrading the courses. And now I think this coming year, another thing that we need to do is, I mean, we’ve talked about this, like making, not just Formula One, but also like scaling out like factory level car production. We’ve been doing a good job of like pressing

like just trying to continually level up the quality of the courses. And now we need to expand also the rate of production of what we figured out how to do, which of course you can’t really do that until you know what you’re doing, right? Until you get the prototype site, get the V1s, yeah, then you kind of scale it out.

Alex Smith: (42:40) As Jason said, it’s like, often said, it’s like, need to transition from being like a workshop to a factory. Like this is, this is what’s happened. This is what’s going to happen in 2026. And we’re having conversations with this, with this, about this almost every day. Now, how do we transition from this kind of workshop setup where it’s me and a guys. I’m just there creating all these courses. It’s super fun, but it’s like, that’s great. It’s gotten, it’s gotten us to where we are now, but if we’re to hopefully become a truly household name.

then yeah, we need to, I obviously to stop being, being that guy, being the workshop man, the workshop man and be like the factory manager. So how do we, how do we go about doing that? Without dropping quality. Exactly.

Jason Roberts: (43:17) dropping quality, continuing

to innovate. You know, and that’s a tough ask. We start, tried to do this. mean, I initially brought up this conversation and it was like November, I think of 2024 was the end of 2024. And I was like, we got to get machine learning cores and we got to speed things up and we got to go. And, you’re, and I was kind of, I was sort of going through sort of what I was thinking target wise and you were just like.

what, you know, and I was like, we have to go and you’re like, well, can I hire more people? And I’m like, yeah, like hire more people. And obviously let’s be careful with our spend, but like we, you we’re, I, that’s fine, you know, hire higher, higher quality people. But I think one of things we ran into is that it’s really, really difficult to find people who are, who have the subject matter expertise and have the pedagogical instincts. How to actually do this thing. We’ve talked about that and I think maybe some we’re talking a little bit more about, cause I like your insight on that.

but that we had the other day. also you just can’t hire that quickly. It’s always a disaster. You can’t just hire four or five people in like six weeks and expect to do something that isn’t like a common skill that everybody knows how to do and expect that to go well. And it did not go well. It caused us lot of inefficiency and lot of frustration. I don’t know. Let’s talk just a little bit about that if you don’t mind.

Alex Smith: (44:39) Yeah. So I mean, one thing I’ve realized, especially over the conversations we’ve had recently is that, uh, I mean, it’s fine to scale and try and transition from workshop to factory by hiring more people and everything, everything you say is true. You need to be very, very careful about how you hire. So you don’t hire on mass. Um, you know, be very, very stringent in your hiring procedures, make sure they’re doing, know, make sure any potential hires are doing well in, kind of like qualifying, uh, tests or, something like that. Um,

But if you’re going to do that, you really need to have the systems and tools in place to just monitor performance. Now, when you’ve got just a ⁓ team of a handful of people, four or five people, content of them, you can kind of keep an eye on what people are doing. that’s probably not the right time to sort of invest in all these tooling and analytics to check that everybody is doing well. But once you start scaling…

As me as a manager, can’t, you know, if it’s 15 people, 20 people all contributing content, I can’t stay necessarily on top of every single one in the same way I could when it was three or four people. we, so you really need to.

Jason Roberts: (45:44) But to be fair, mean, you were fine up to like, a dozen.

Justin Skycak: (45:49) 12 to 15. it was kind of impressive how far you were able to

Jason Roberts: (45:52) We

were always like, we’ve been 15 people or more for years and years. So you were fine. It was when it starts to creature up to like 20, 18, 20, 23, 25 when it’s like, stuff just goes, because you don’t have the, like you said, you don’t have the transparency, you know, about equality, you know, everything where you can easily say, well, this is not happening or these aren’t happening the way we want. You just have to kind of try and pick up on signals of whether stuff’s going on, right?

Alex Smith: (46:19) Yeah, that’s it. And of course, obviously, we’re truly international. mean, my team are kind of based all over the world. And so it’s like, it’s not we’re in the same office. could just I could see that guy is sort of that guy over there is like not doing much, you know, he just seems to be ⁓

Jason Roberts: (46:33) Guys

playing Galaga. He didn’t think we’d notice. But we did.

Alex Smith: (46:37) Whereas in a of completely like, you know, international, 100 % remote setting, this is like, this is even harder. You know, it’s like, you need kind of proper reports. What has this person done today, this week, this month? How is that comparing to other people doing a similar job? And so it’s all these kinds of analytics that need to be in place if you’re going to kind of scale up successfully. And it might not necessarily even be that person’s fault. It’s just like, they think they’re doing a good job. They’ve read the training material. They think they’re doing a good job, but it’s just, it’s my job to have access to that.

that data and go, actually, that guy isn’t doing good, but let’s find out why. And, you know, perhaps just the right, just one conversation is all it takes to boost that performance. But unless you’ve got the data, you just, you just don’t know really.

Justin Skycak: (47:19) Yeah, no, really seems almost like building a software system just made out of people. It’s like, well, like you can’t blame the people for not giving a result that you wanted if you didn’t like write the code correctly, right? It like, just like, you need to invest time into like setting up the right abstractions, the right efficiency, the right like job allocation, work allocation, everything and monitor that that’s getting the results that you want. Yeah.

Jason Roberts: (47:49) That’s something that I was talking to Justin about a couple weeks ago is thinking about Mathcademy as a machine that you’re trying to optimize. It kind of reminds me of that joke that was that a tenor mathematician, Eridish, would say that mathematicians are machines that turn coffee into theorems or weak coffee into lemmas, something like that. so I was like, by that similar analogy, we’re a machine that turns

money subscriptions into math education, into mathematical outcomes, right? To successful mathematical progress or educational progress. And it’s like, okay, how do we optimize this machine? So part of it, we have a lot of analytics that we’ve built over the past few years that Justin Particula has built with a lot of these tools that you use to analyze pass rates and where kids are struggling and everything. So you can really fine tune, be alerted to say, okay, well, we’ve got some low pass rates on some topics.

What are, where are those, where are those, where are the, where are they failing? What’s the number, how to fix it? And then you can use that to fine tune. so to continually upgrade content, which you’ve, you know, we’ve had some of these tools around for years now, which has really, really helped. And, but now we need to extend that to, we need more metrics about how the company is doing. I think, you know, you know, we have certain amount just in terms of like, you know, we look at our Stripe dashboard and go, okay, what’s the

Justin Skycak: (49:14) Yeah, that boilerplate.

Jason Roberts: (49:15) Boy, we need more. We need a lot more. But we also need, I think, a lot more because by far our biggest cost center is content. And it’s like, okay, are we getting our money’s worth? Are we moving as fast as we can? Like, when is stuff behind? Is it on schedule? You know, whatever. And so what we really need and which we’ve started working on is a lot more. ⁓

Metrics into understanding how this process is going, you know, how it’s going overall how a course is progressing how individual people are doing You know how how efficient our spend is because it’s like okay if we’re create You know two three four or five times amount of courses that we did in the past and they without dropping quality like we have better have a really granular level of insight into every dollar that we’re spending on this

Justin Skycak: (50:06) Yeah, it’s basically like we’ve done a good job of this for students for like optimizing their learning efficiency, like analytics on the knowledge graph and everything. So everything that we’ve done to the learning process to make that what, like four times more efficient, we have to do it to ourselves also as an organization, make ourselves four times more efficient. But everybody through who’s doing a particular workflow create like a, almost like a step by step, okay, here’s what the workflow is. We can measure performance every

step of the way that the cattle shoot, just like the lesson and have it almost gamified a little bit in terms of like, ⁓ like production measurement. Like, okay, who’s, who’s doing how much each week, how many work units are you, you, are you putting out? And is that, is that good or bad or how, does that compare to what everyone else is doing?

Jason Roberts: (50:57) What’s the error rate? What’s the error? You know, how was error rate on the quite really with. You know, it’s OK. Well, you’re turning out a bunch of questions, but they’re not very high quality or something or whatever the problem is. So it’s like, yeah, so we have to have a really good holistic measurement, not just of the quantity, but just of the quality as well. So creating a high quantity of mediocre quality is pretty much a waste. So.

Justin Skycak: (51:01) quality of the work.

Jason Roberts: (51:23) But of course you can’t have, it’s high quality, but I’d do like one question an hour. It’s okay, well dude, that’s going to take us years and years to create a single course. That’s not, we have to be moving at a reasonable pace. So yeah, so building out the metrics for like our inputs and our outputs as a company and the outputs being the quality and efficiency of the learning, the engagement of the users, the retention of subscriptions and things like that. But also.

how our course development process, how efficient that is so that we can create the courses everybody wants. Because Sandy will tell me, okay, when can we have this course, when can that course? Because people are just asking for everything. Right? It’s not like, oh, we got a lot of the courses. don’t have a fraction. Because it’s not just what’s going on in the US. It’s people in other countries want stuff. Or people have different curriculums. They’re like, oh, do you guys have the, what’s it called, the GSE, whatever?

Yeah, the stuff like that or IB or whatever, different AP courses. AP stats, same as Benescom. You guys need to do, we got to do AP stats. what about you? I mean, there’s just, it’s just an ending.

Alex Smith: (52:31) Yeah. Just to circle back a little bit to the, ⁓ to the, you know, the, the, it’s all about having like a leaderboard for like content developers and metrics, which allow us to sort of track who’s, you who’s doing what, who’s performing well, who needs more help, that kind of thing. You can actually make that really fun. know, I mean, you’ve got to make sure you’re optimizing for the right stuff and that it’s fair. But if you get to that point where every, every, ⁓ contributor, every, every, ⁓

SME subject matter experts contribution has been measured effectively relative to everybody else. You can kind of turn that into a game and we gamify the curriculum is somewhat gamified. You know, there’s, leaderboards and stuff like that. You can do the same internally and it can be really fun. And the people that you really want to have around will find that motivating. It’s like, yeah, you know, like, you know, you turn it into a game, have competitions, prizes for people that perform.

Jason Roberts: (53:21) I want to incentivize and reward people who are doing a good job. You know, and the people who are kind of dragging ass and are like not, you’re like, okay, well, you know, it’s like that Sandy and I were joking about the, about the scene from office space with Jennifer Anderson and she was her boss. And she’s like, and he’s like, you know, I to a conversation about your flair, you know, because she was at a restaurant where they can only wear so many buttons and stuff on their, on their uniform. And then she’s like a server at a restaurant, right?

And he’s just like, so, so more fired. He’s like, well, some people just, you know, some people like to do more and we encourage that. She’s like, so I’ve enough. He’s like, it’s like, it’s a good, ridiculous thing, but it’s like, yeah, you want to say, look, you know, like here’s a minimum level of like, what is, what is acceptable and you should know what’s a reasonable level, right. Of expectation, but also the people who are going to kick ass. It’s like, okay. You know, guess what? You’re getting a bonus this week. know, XP.

You know, go, dude, it’s awesome. You know, you know, I’m not, I want to be fair. Now we want to help people do a good job, but then I’ll reward people who do a great job and so that they stick around and keep doing a great job for us. So it’s a win-win in both ways. So I think, yeah.

Justin Skycak: (54:37) Yeah, this also gives like just more clarity to on like expectations of like, okay, like, what are we doing? What is our goal here? What are you trying to do? What are you trying to optimize? And it’s like, well, okay, optimize work production relative to a quality threshold. It’s like, okay, well, there we go. Now everyone knows the game, let’s play it. Yeah. Provided that we properly align that game with and all the metrics involved in it with the true mission that we’re trying to accomplish.

Jason Roberts: (55:06) Yeah, I’m excited about it. think well, so yeah, I mean it’s like we haven’t rolled it out We’re still kind of fine-tuning and we’ll run it kind of just in private so we understand until we feel like okay This is truly I think this is I think this is right. I think these numbers are fair I think these Calibrations and weightings so like how long certain tasks should take and whatever within some threshold are good are on point then then we can you introduce it, you know, course again We’ll be we’ll be fair with everyone. It’s not like gonna be like, my god, you blow this you’re fired

It’s like, okay, well, you can we should build a little more here. Let’s talk about your process like we help you what I want to do is help help our content developers anyone who are dragging like help like let us understand your process and how we can help you be more efficient because you should be able to move at a faster rate. What are you what are you what are you doing stuff like that? Anyway, it’s gonna be interesting. But I think all of this streamlining of our process the metrics all this kind of stuff is gonna help us

get a lot more courses out in 2026 than we’ve ever done before, probably by quite a margin because we need to because people are getting annoyed. Like how long is it going to take to get these things out? the answer is, yes, we need to go.

Justin Skycak: (56:18) Step on the gas. Yeah.

So that sounds like that was kind of like the year in review and year ahead on the content side of things. Is there anything else that Alex or Jason, you guys want to talk about the content side of things here? ⁓

Alex Smith: (56:36) I’ll just mention a few things briefly. mean, obviously once we really get these processes nailed, the sky’s the limit in terms of courses and what courses we can do. But certainly I think, you know, just things like finishing off the sixth to eighth grade curriculum, I think is sort of relatively low hanging fruit and something we need to do because we’ve got fifth grade, we’ve got pre algebra, so where’s the gap? you know, obviously some schools are using Math Academy at the moment and…

for more schools to use Math Academy, that gap needs to be filled. So that’s important. I think we’ve already mentioned like the ACT stuff to complement the SAT. I think that’s going be great. At some point introduce more, we’ll start probing into like more proof-based university level courses, abstract algebra. Probably need to do, we need to have some kind of freeform proofs in the proof-based math courses. So we’ll probably go back to methods of proof.

get some freeform proofs in there, because that’s our of our base course for the proof-based stuff. And then use our experience and knowledge that we gained from that to move on to abstract algebra after that real analysis and stuff like that. And I also love a course on complex variables. I think that’d be really cool. Probably not quite as challenging as abstract algebra in terms of, we don’t need much new tech for that.

Jason Roberts: (57:51) It’s

easy to move a line of like the difficulty of multivariable calculus or differential equations. definitely. Because it’s purely computational. It’s complex variables, not complex analysis. So our approach, which I think makes sense, is that we kind of take a concrete version of course before you do a proof version of course. So linear algebra doesn’t have any proof. So if we had a second linear algebra course, it would be entirely proof-based.

Because the problem is when you do a proof-based course and the students don’t actually have any real intuition on how these operations work and they haven’t done any problems and they’re just proving stuff about objects that they don’t really understand deeply, you’re not getting as much out of it. The student isn’t getting as much out of it. They’re struggling. So linear algebra, you know, and then if one, linear algebra two would be the proof-based version or something, I don’t know if that’s what we call it, but then the second one, or advanced linear algebra might be.

proof version and likewise I think you do complex variables and then you’d have a complex analysis. But if you were like a engineer physics type person you would say okay I’ll just take it don’t need the proof base I’ll just do complex variables and I’m good to go.

Alex Smith: (59:00) Yeah. Yeah. Just one more thing. I’m sure it is obviously. Yes. We mentioned like machine learning. I mean, the machine learning track potentially could go really far, you know? So yeah, definitely a second machine learning course would be on the cards. Yeah.

Justin Skycak: (59:03) ⁓ you go ahead.

Yeah,

you know, I’m really excited to get like the more advanced versions of a lot of these university courses, like advanced linear algebra, and then not just complex variables, but like complex analysis and stuff. Because I think, well, at least it seems to be confusing for a lot of people ⁓ who are not so familiar with university course offerings. They don’t understand that there are levels to things like linear algebra, complex variables, complex analysis, even stuff like

Sometimes I would get asked like, well, how come your, your calculus one course, it doesn’t have like a Epsilon Delta limits in it or like, or, or, or proofs like, like that. And it’s like, well, like that’s what you do in real analysis. It’s not necessarily in a calculus one course, unless it’s like a sort of honors calculus at like an elite university, which is really like more like a real analysis course. They just call it calculus because they’re

real analysis course is actually like advanced real analysis, that sort of stuff. So there’s kind of a fuzzy labeling on these things. So I think once we have kind of the second more advanced version of each of these courses out, it’ll be very clear. Like, OK, here’s what linear algebra is. Linear algebra, advanced linear algebra, that’s more like Axler’s linear algebra. And know, Axler even says in his book, like, is my linear algebra course is meant for somebody who has

basically already taken the first course, who has already had exposure to it. But a lot of people just kind of did not read that or just don’t want to believe that like, well, you don’t just jump straight into Axler unless you’re just incredibly gifted and can backfill all the missing prerequisite knowledge on the fly. But for most students, yeah, they need to kind of go up these levels one rung at a time.

Jason Roberts: (1:01:07) You know, it’s funny, you’ll hear these sort of like ridiculous conversations in places like Hacker News when these books about these, you know, discussion math books and they’ll say, oh, well, I think this is the best, the Axler or whatever. And they’re not really, you know, they might be have some fond memories of this more advanced course, but they’re not really appreciating how much foundational knowledge a student needs to have to come into it to be successful. As they’ll just say this. And so if you don’t know any better and then you try and use that book, the first one, it’s a total

flame out, you know, just like, I’m just, I’m just dumb. I’m not smart enough to do that. It’s like, no, you listen to somebody who’s just talking on the internet, which is always dangerous. People just say stuff, you know, they’re not personally accountable to whether it’s going to work or not, or really even know what they’re talking about. And then you listen to it you’re like, I guess I should do that. But most universities don’t even approach it that way because that just doesn’t work. it’s, a bad strategy, you know.

Alex Smith: (1:02:06) says

that it’s just the curse of knowledge isn’t it? like, you know, I remember in Mastering this advanced linear algebra, that was really cool. And that’s the book we use, you should take it. like, as you say, just no appreciation for how they actually probably got to that point in the first place, which is to grind through things like raw operations and things like that, know, normalization.

Jason Roberts: (1:02:27) I

can buy you some things like that.

Alex Smith: (1:02:29) That’s

you have to do first, you know, and I say some people can go straight to Axel’s book, but they’re pretty rare. They’re very rare.

Jason Roberts: (1:02:37) Yeah, I mean, because so I studied, you know, math at University Chicago and we didn’t, if you’re a math major, after you took real analysis for a year, you took a year of abstract algebra. And in the abstract algebra course, they didn’t have, you didn’t take, there was no linear algebra course, right? The linear algebra was embedded in the abstract algebra sequence and it was mostly proofs. And was terrible. I mean, it was just…

I remember our textbook was at Hungerford, which was an abstract, it was a graduate school textbook. It was like the yellow, like the ones behind your head up the top of the shelf, the yellow graduate text. That was our textbook. And it was just so much harder than it needed to be. yeah, ⁓ I guess if you’re dealing with some absolutely brilliant students and they just work like crazy and they’re working groups, you can kind of.

get them to fight their way through it, but it’s so inefficient. It’s so painful and it’s just unnecessary. So, but anyway, yeah, I, is, that’s not the way that is not the way it is not. And, and, and so what we do is take a concrete, straightforward, let’s get your, your, your, your skills, your intuition up. And then once you have, you really understand how to do all this calculation, you really understand what eigenvectors and eigenvalues and row echelon form all stuff is just like.

Instinct to you then we can make proofs about it and you’re in writing proof about that is not be harder than writing proofs about Divisibility and parity because you just understand it intrinsically and then it’s just a matter of You know thinking the insight having the insight and through the proof. But anyway

Justin Skycak: (1:04:16) Yeah, as you always say, intuition comes from the repetition, right? And then the proofs is really just a demonstration of you. I mean, it’s really represents intuition, but that’s not how you absorb intuition. That’s the product of intuition. Intuition is not the product of that.

Jason Roberts: (1:04:34) Yeah, yeah, it’s it’s a that’s so frustrating when people say, I just just give me the concept intuition. It’s like, can’t just get you the intuition, you know, intuition is a product of a lot of hard work. You know, that’s where the intuition comes from, you know,

Justin Skycak: (1:04:49) analogy

I sometimes like to make is like, suppose that you had a book of life quotes, right? From your hard-won experience in life and you have a kid and when the kid is like five, 10 years old, you hand them book and you’re like, let’s read this together. Like they have no idea what you’re talking about. You can be completely right. The quotes can, I mean, in your mind, they can represent so much valuable knowledge, but like…

The kids are not gonna know. even if they say like, I get it.

Jason Roberts: (1:05:19) They don’t know. Even someone in their 20s. Yeah. Like, you know, it’s funny, my middle daughter went for a year, she went to this sort of, you know, kind of fancy, progressive private school. And they were, she was in ninth grade and they were reading Adam Smith. I was like, she doesn’t know barely little supply and demand. She doesn’t know what, she doesn’t know about economic free markets. I mean, she has no intuition for how commerce and economies and…

find me any of that stuff works and you’re talking about, you know, you know, the Adam Smith level theory of, of, of economics. It was just so, it was like performative. It was like to show off to the parents, like how sophisticated our kids are. It’s like, this is just, it’s just dumb. And I mean, I even sometimes think of like stuff like reading Plato and all the stuff that we did at Chicago. And I think that stuff really is more useful when you’re like your thirties.

I have some mileage on the road come back to stuff like what is truth really? You know like what what is fairness? What is justice? Yeah, you have to see a lot of examples of of these things people struggling to come to any agreement consensus on these things and they can come back like let’s go down to the basics and you know, I always feel like it but just not how our world works and but you’re right you can’t just you know like trans take this like

consolidated compressed knowledge and put it someone’s head and it makes any sense because it just doesn’t. You know, they can mimic it and they can just repeat it to you. know, that’s amazing. They know this stuff. It’s like they don’t really know it. You got to, I mean, all these life lessons and stuff, you got to suffer. You got to make lots of mistakes. It’s like when you’re an adult or a parent and you’re teaching your kid, your teenager and you’re like, listen, dude, like, bang, bang, bang. And they’re like, okay, whatever. And you’re just like.

You’re going to learn the hard way. know it kid. And they learn the hard way and you’re like, so how’d you enjoy that? What did we learn from this? Had some recent, very recent conversations with like my college aid son about some life lessons. And I was like, you know, you can talk to a very, very bright, open-minded kid who respects what you say, but they’re just, don’t get it. They can, you have to go suffer. You have to make mistakes. You have to get some mileage on the road. And that’s the same with math, you know, as anything.

Justin Skycak: (1:07:39) You got to get the details, the microstructure before you can abstract it. Yeah. Right. Otherwise you just end up feeling like you have just been pushing symbols around in some, some sequence of logically accepted conventions. And you’re like, what’s an eigenvector? What’s an eigenvalue? I don’t know, but it’s like, it has this symbolic property that like helped me complete this proof. So I guess that’s what it is. Well, yeah, you don’t really know what it is.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.