Link to Podcast

What we covered:
– Building a knowledge graph is like city planning & road construction. Too many prerequisites leading into a single topic creates a cognitive traffic jam.
– Elegantly rewiring a live knowledge graph: the evolution of our tooling and automatic validations. How to avoid staging servers & migrations and NOT have it blow up in your face.
– UI work takes time and adds complexity, so we spend it on the customer. Internal tools are almost entirely command-line; clickable buttons are for customers.
– Justin's transition from research coding to real-time systems. He started with mathy, notebook-driven quant code and had to learn production engineering the hard way. Once he did, it was a massive level-up.
– Alex's plan for dealing with "content papercuts" - small issues that pile up. Inspired by Amazon’s "papercuts team."
– Our upcoming differential equations course, the last course in the core undergrad engineering math sequence.

Timestamps:
00:00:00 - Building a production-grade knowledge graph is like city planning and road construction
00:07:26 - Elegantly rewiring a live knowledge graph: the evolution of our tooling and automatic validations
00:24:47 - Justin's transition from research coding to real-time systems
00:44:51 - Alex's plan for dealing with "content papercuts" - small issues that pile up
00:58:02 - Our upcoming differential equations course

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

Link to Podcast

The raw transcript is provided below. Please understand that there may be typos.

Jason (00:00) You know, it’s, ⁓ you know, it’s interesting about sort of planning and architecting the, the knowledge graph, which is Alex, you’ve spent a better part of the past, I don’t know, six years, seven years doing. Yeah. But I mean, we, yeah. And, ⁓ but it’s like, one of the things is you almost think like each node is like an intersection. You can’t have too many roads, you know, coming in of, you know, into a single intersection. Otherwise it’s a mess. You typically don’t want.

four occasionally have a five or city has like five points right. I recall that was kind of a pain to get in there and it’s like the light situation in the call the crosswalks.

Justin (00:40) Yeah,

and you try to avoid those intersections because there’s such a headache.

Jason (00:43) Right. So go, okay, that’s the only way we can get to the, you know, the, uh, the zoo is we got to go through that five points thing and Lincoln park, you know, you actually know where they are. They’re, you know, that time just like cool places, but it’s like, it’s not efficient. Right. I don’t know. But, um, so Alex, like, you know, with that analogy in mind, I mean, when you kind of think of heuristics, you’re like, okay, when your spider senses start tingling, when you get too many prerequisites is, is like, you don’t, it’s like two to three.

Justin (00:56) You got emotional scars from it.

Jason (01:13) One to two is great, three is okay, four, like what point do you go, this is, we just can’t feed too much into one. That’s causing a type of a cognitive congestion. What would you say?

Alex (01:26) Yeah.

Yes, I think in terms of direct prerequisites, I usually say that three or four is probably I start getting feeling uncomfortable when it’s anything more than that. But we have direct prerequisites, which are in the knowledge graph. They’re sort of like, you know, sort of like it’s like one edge connecting the topic to its prerequisites. But we’ve also got this other thing called, as I mentioned before, key prerequisites and each each knowledge point within the topic, each each topic is broken up into three or four knowledge points.

And each of those knowledge points has its own set of prerequisites, which are key prerequisites. Now, some of those key prerequisites might also be direct prerequisites, as in they studied it not that long ago. But some of them can be actually quite detached from it. It might be three or four jumps in the knowledge graph, maybe even more. So you’re bringing in knowledge that students studied at some point in the past. And when those start stacking up,

⁓ is I think is when my spider sense is really start tingling like every single knowledge point needs like three or four key prerequisites. It’s like, ⁓ this is this is feeling like cognitive overload.

Justin (02:31) ⁓ probably because the direct progress is it kind of like as for the topic as a whole, but the key prerequisites are for each particular knowledge point. And that’s really the pressure point, ⁓ as, the student moves through, right? It’s one, one knowledge point at a time. So you want to measure the price, how much cognitive pressure on each knowledge point, you know, what’s, what’s interesting is that you said the three to four, ⁓ prerequisites. mean, when you, when you look at, ⁓ working memory, ⁓ literature, ⁓ somebody can hold about.

⁓ four chunks of information in working memory. That’s about your capacity.

Jason (03:07) that

coincidental correlative or know, with a term of, you know, it’s interesting. So, I mean, one thing we, this knowledge points are something more we talk about internally. I don’t know how much we talk about this publicly, but every topic is divided into typically three to four knowledge points. In extreme cases, you have two or five, but typically it’s three is the ideal, sometimes four. And what that means is the student progresses through a topic.

in the most basic vanilla version of that topic. Just get some wins, just get basic concept down. And then the next knowledge, the second knowledge point might add a little bit of a twist. Okay, now we’re gonna do negative numbers or odd exponents or whatever, something that, and like, okay, they do it. And then the next one is like another twist. Okay, now we’re using fractions or radicals or exponentials or something about it that makes it even maybe a little.

more nuanced or tricky and the fourth one might even pull them all together. So it’s sort of like this, if you’re teaching a student to do something like I’m solving linear equations or I’m taking derivative, you start with the most basic and you’d progress. And once they do not go to knowledge point two until they can at least do two problems correct in a span of five in the current knowledge point. then you know.

Okay, you got couple. Okay, good. All right, that’s good. You did that. You took a derivative. Okay, three X squared. Okay, that’s six X is driven to that. Okay, now let’s try it. What if it’s, you know, two X to the negative fifth and I go negative. Well, negative five times 10 and then, you know, whatever. then, you know, I mean, so you kind of build confidence and you do that. anyway, so for, for, for our viewers, listeners, whatever it’s, it’s, ⁓ knowledge points is just a way of separating and creating a step, a sort of a ladder.

within ⁓ a single topic. And individual knowledge points can have specific ⁓ pre-keeper requisites. So if you were like, say, OK, the third topic introduces radicals somehow into this situation, it’s like some prerequisite might be algebraic operations with radicals or something, right? So something like that. ⁓ But yeah, so it’s interesting. mean, there’s probably a lot we can talk about.

postpone it for some later shows about, you your methodology and what you’ve learned and all the heuristics you’ve developed for like, how do I create a knowledge graph that’s effective?

Justin (05:33) Yeah, this is not just based on your gut. mean, part of it is based on your experience as an educator, but part of it is also just looking at cold, hard numbers, analytics on this thing. It’s like you’re like the city planner with all the traffic routes and everything, but the nice thing is if there’s a five-way intersection that’s just not working out too well, you can just like pick up the roads, like rearrange it. ⁓

Jason (05:57) Yeah, that’s a great analogy. You can imagine you’re like the head, you know, city planner, right? And you have some young guns come in and they’re planning. You got some new part of the city that’s going to put some roads and they and they and you initially come in and you’re like, that’s not going to work. Like, what do you mean? Like, OK, you guys, got this exit to from a highway and then we got a subdivision and then they’re like, like, you know. This so you would say, look, so you would instinctually know, having spent years doing this.

where there are likely going to be problems. But then at the end of the day, what you do your best, and then we start getting data in and it’s like students are struggling with this topic. We have a pass rate that’s not at our benchmark of 94, 95%. It’s 75 % or something like that. What’s going on there? And then you go and Justin had built a bunch of analytics, bunch of tools that you can use to find, to sort of inspect and go, okay, where are students struggling? You’re struggling at knowledge point three.

They’re fine up to knowledge point two and knowledge point three. And so then it’s like, what’s the problem? Like, what is the problem? typically it’s one of two things. need a, we need a, we’re missing a knowledge point. Sometimes it’s like further elaborating on something, including a graph or more detailed description or something. But a lot of times it’s inserting another knowledge graph. Cause it’s like having steps, except you’re missing a step and people have to step two steps up and it’s just too much. Or sometimes it’s splitting it into two topics. Right? mean, what, I don’t know. It’d worth talking about that for a minute.

and how do you

Alex (07:26) It’s funny, actually, I remember the first time that I said this, Jason, I think I really need that it was I think it you at the time it wasn’t Justin. I said to I think I really need a tool for splitting topics like into two like this topic is overloaded. I need I want to be able to split it there. I remember you were like super busy at the time. Well, I really hope that this is worth it because it’s gonna take me all weekend to do and I must have used the splits or I reckon like 500 times.

It’s hard for me to say because that is the solution to so many ill. Just break this damn thing into two topics. They don’t have to do Euler’s method, one step, multiple steps on the same day in the same lesson. Let them do one iteration first, split the topic. Multiple iterations can be, once they demonstrate mastery of that and you have sat in their brain for a day or two, then they’re ready.

So that’s been incredible. That is like a savior because yeah, like I said, we spend a lot of time looking at data, like student performance data, and we can see exactly where students are struggling. And like you say, it’s often, you just kind have to use your experiences in Educator to understand, what’s the most likely problem here? Are they getting confused because the diagram doesn’t work? But often it’s just try to do a little bit too much in an individual topic. Just split it.

Everyone’s happy, usually.

Jason (08:52) It’s

like you can explain anything if you bring it a small enough like, you know, into small enough step. It’s truly trying to do too much. It’s usually they’re trying to take too big of a jump and then people just fall off. It’s like, you know, used to than stairs. I used to think of it as like rocks placed across like a creek or something. And it’s like, okay, well, if they’re they’re three inches apart than grandma can, you know, where he’s what grandma can do it. If it’s like eight inches, it’s like, well, you know, my mom and dad can do it. Okay, now it’s

a foot and a half, it’s like, well, I can do it and stuff. And now it’s good point where it’s like, well, my son can do it. Three feet, you know, it’s like, that’s what happens. It’s like, well, only super gifted, brilliant people can do it. It’s like, because the pedagogy is so bad and the jumps are so big, you need to need someone who could do a standing broad jump of nine feet to make it right. Otherwise they’re landing in the water. And it’s like, OK, why rather than writing textbooks that have these

six and eight and 10 feet jumps every two or three pages. And it takes them with incredible athletic ability or incredible mathematical aptitude to make those jumps. Otherwise they’re just like, man, I just, I don’t know. I’m like falling into the river and I’m floating down the creek and like, wait, this sucks. know, for, you know, essentially, so I, so going back to that split tool, so it’s been a long time. So I’m trying to remember, but I remember initially, cause this is back, we didn’t have many,

It was mostly just the school program and maybe and we would like split live topics. We would lock a topic. Well, first we didn’t even lock topics and I’d be like, Alex, make sure you don’t split it after because we had because we did at that point. that point, we had ⁓ one beta tester, one person who used to be in the school program and she had moved her family moved away and the dad is like

Alex (10:21) Yeah, I play it.

Justin (10:26) Yeah.

Jason (10:46) What am going to do, Jason? They don’t teach calculus in eighth grade. I need to do math. We had them using the system and she was like on the East coast. And I was like, okay, come 8.39 AM Eastern time. Cause sometimes she’d do it in the morning before she would go to school or something. And I was like, you can’t be splitting topics. You got to have it done and split by that time. Cause you’re in the UK. So you only had so many hours and our students in the California are doing math.

up until as late as 11, sometimes midnight Pacific time. And so you had this relatively narrow window in which case you had do it. And you were kind of stressed by that, right? You’re like, right? I’m already being a stressful thing for you.

Alex (11:27) I mean, obviously, there’s, there’s times when when things are extremely raw, and things can break. You are, it’s a stressful experience to know that you’ve done something which has caused some kind of break. And that happened in the few. Yeah, I mean, it’s much, much, much rarer now. But it’s, but yeah, obviously, in those very, very early days. Yeah, it’s, it’s kind of a stressful thing. If I do this,

Justin (11:55) just kind of flying by the seat of our pants back then with the like, it doesn’t look like any students are working on. Okay, you can you can do it, but have it done quickly. So it’s like that only I mean, that works when you have like just a handful of students, right, but eventually you you just get into a state where like, there’s never a break. It’s like you’re the city planner, right? And now there’s never a break in traffic. There’s traffic at all hours. There’s like, and it’s like now

Jason (12:23) world now, so.

Justin (12:24) Yeah, right. Exactly. And so it’s like, really, there’s a lot of like, a lot of stuff that has to happen behind the scenes now. It’s like, okay, we lock a topic. Well, you clone a topic, get it all prepped for the switch. And then we were like, okay, it’s ready to switch. We lock it really quickly. Do a nice switch onto the new topic, swap everything out and then unlock it.

Jason (12:49) was a lot of… It went through progression. Initially, there was no locking. Like a student could literally click on it and you were like, hurry, get it done. We were like, make sure you have it done by so… And then we, then Justin and I worked on a thing where you could lock it. Like if a student was using, or they try and submit an answer and it was locked, it’d be like, ah, we’re currently, this is under construction, sorry. And so the student would be like, that’s the worst case. I mean, we tried to avoid that too, but you would, you would lock it.

And then you would do the surgery you needed to do, whether it was splitting or adding an actual knowledge points or moving questions around or whatever, and then unlock it. And then we’d have the students lesson or whatever, update it as necessary. But then we’re like, that was fairly limiting and stressful. And then we’re like, okay, and we’re just sitting around thinking like, okay, how are we gonna do this? And we’re like, all right, I think what we need to do is we gotta clone this sucker, clone everything, the question, the knowledge points, all the…

all the related records to this topic. And so that you can do whatever the hell you want. And then Justin built a whole suite of command line tools that you use that you would like clone it. And then you do all the stuff you want to do and the team could work on it do all the stuff they wanted. And then you would build some tools where we’re then, I don’t know what’s it called, what’s that? A knife switch would just insert it in, it would replace it, right?

Justin (14:09) Yep. Yep. Yep. That’s right. Yeah. Do everything really fast. Right. Cause the issue that we encountered with, with just locking is, like, okay, well in theory, like, okay, that’s in theory, it’s fine. As long as you, you get it back up like relatively quickly in the same day or something. But once you get enough students, you get to a point where it’s like, we got now, now we have one, two, three, five, 10 students who need that topic to make any progress in their course, like their bottleneck.

and they’re exactly.

Jason (14:39) We’re gonna get emails. We’re gonna get emails from Xamarin.

Sandy’s gonna be like, what is going on? I’m getting all these emails. You know.

Justin (14:45) Yeah, yeah, right. now we’re at a stage where every topic is needed by every student. At any moment in time, there’s enough students needing every single access to any single topic that we just we can’t take anything offline for even a couple hours. So ⁓ yeah, so that’s how the Knife Switch tool came into being.

Jason (15:09) I

think that was some of the very first command line tools that you built for Alex. And now this hundreds of command line tools or something, each one is like a Unix program that has all these options. Like Alex has this huge tool box.

Justin (15:25) Yeah, yeah.

Yeah, yeah. Well, I think it started out actually, the first command line tools that I built were validations to make sure just that things are all set.

Jason (15:39) Because

whatever he did didn’t break, didn’t break anything.

Justin (15:42) Yeah, yeah, stuff, stuff like that. And then yeah, knife switch came after that, but yeah, we’ve got hundreds of, of command line tools and, ⁓ yeah, yeah, actually we’ve even got like maybe like a hundred validations that that run periodically every so often to make sure the system is in a stable state. And each one of those is a command line tool. There are any issues that’ll spit out the description of the issue and stuff.

And so I’ve got a validator that goes through, runs those, collects the output, pastes them to Slack so it can notify us if there’s any issues.

Jason (16:13) mean, that stuff doesn’t, we don’t have too many problems these days with with knowledge, any of these things, right? I mean, this stuff is pretty

Justin (16:21) No,

but we used to, Alex, do remember how stressful it was to tweak the prerequisites and worry about introducing a cycle or just some kind of invalid graph connectivity that then the model just falls over and dies?

Alex (16:34) That was the worst. Yeah, cycles were the worst. And it did happen a few times, I make some kind of change. And then all of sudden, we start getting complaints. There’s no validation. There’s no like validation tool that’s kind of spitting stuff out. just stuff isn’t working. It’s like, we think there’s a cycle in the graph, what’s gonna happen? What’s going on? So I like, so that’s kind of like retrace my steps. Okay. Yeah, I think it might be there. So go back and tweet the prerequisites is all five. That was extremely stressful. Because I remember exactly what you did to

Justin (17:01) It’s like playing Russian roulette knowledge graph edition. You just do it enough times and like something breaks and like it’s the whole thing. And yeah, so now we’ve got every single graph change that you, that you put in, it gets validated to make sure that, that it can go, that like, that there’s nothing, no mistakes being introduced into the knowledge graph unintentionally. And I’ve even got like a self healing, self repair in the models that if there are any, any issues, it does its best to try to keep on functioning despite.

Jason (17:30) Yeah,

right. that well, oh, first of one thing I wanted to say is for people to cycle is it’s like it’s like A’s a prerequisite of B, B’s a prerequisite of C, C’s a prerequisite of D, and then D’s a prerequisite of A again. it is. So that’s a cycle in a, you know, in a graph. I would call an acyclic directed graph. It’s we have a graph. It’s it’s it’s not like a graph like I’m graphing a line. It’s a thing of nodes and edges, things all connected to knowledge.

Justin (17:57) Yeah, to connect the dots sort of thing. Exactly. Exactly. So it’s like a stepping stone, right? It’s like the stepping stone somehow go in a circle and you’re like, wait, I’m supposed to be crossing this river. Why are the stones going in circle? Obviously it’s not what we intend to do, but when you got so many, like when you got thousands of topics and tens of thousands of prerequisite edges, it’s easy to sometimes just, you know, type one number in wrong. And now you’ve got a prerequisite pointing to the wrong topic.

Jason (18:22) It’s mean, mistakes,

right? Alex types in. Exactly. Oh, it’s 1, 2, 3. It’s topic ID, 1, 3, 7, 5. And he writes, he accidentally types 1, 3, 7. And they didn’t get the.

Justin (18:32) Yeah,

so got to catch that. Yeah, you got to have automated catches of those things because it’s just, there’s some amount of human error that’s just, you can’t avoid it ⁓ in the aggregate. But yeah, and that cycle, that’s just one example of an invalid graph connectivity. There’s actually like maybe four or five different types of these things. There’s gaps between courses. There’s a number of things. ⁓

Jason (19:01) There’s topics where you could have like, here’s a course of 200 topics and there’s a topic that’s a prerequisite that sits in between two topics that’s not in the course. Right?

Justin (19:11) Yeah,

exactly. it’s like you need it’s like the you need there’s an intermediate stepping stone that is not included in the course.

Jason (19:21) And that’s easy to do sometimes. I back in the day before we had all this tooling. ⁓ Yeah. But I remember the reason we built the tools, because Alex, you’d be like, well, it’d be really useful if I had this and really useful to have that. And when you have to build a whole UI for it, I mean, it’s 10 times the work of just creating a command line tool.

Justin (19:30) Yeah.

Jason (19:51) Right. Because there’s all these like, well, in the base case, you just write a button here and this and that like, well, what if they do this and what if they do that? And you got to check this and you got to show that. And it’s like, it’s just blows up and this becomes a whole thing. And meanwhile, I’m just trying to build all this stuff for our actual customers, students and parents and school, whatever. And you’re like, Jason, I’d really great if I had tools, so stuff. And I’m just like, dude, I mean, how badly do you need this? You know, and so I was like, all right, Justin.

Could you spread a command line tool? Because Alex is the only person who’s going to use it. So it doesn’t have to be this.

Justin (20:27) interface.

Only customers get clickable buttons. Alex, you’re going to have to type the text into an output file and just read it there.

Jason (20:39) But it probably works for you though, right? Having a bunch of these tools. Because the thing is, if you have command line tools and you’re fast with it, it’s actually, all the Unix heads will tell you, it’s like they prefer it, right? It’s so much faster if you know that your fingers are, even muscle memory in your fingers to do all these command line tools, as opposed to clicking crap around, right? mean, it, how does it, how they work for you?

Alex (21:03) Yeah, I mean, when you first suggested the idea of having all this stuff as command line tools straight away, I jumped on the idea. Yes, that’s great. mean, in a past sort of life, if you like, I kind of like did a bit of like, you know, Linux system administration and stuff like that. I was saying I’m FTP servers and HTTP servers and things like that. I kind of got, you know, when I was doing my PhD, I kind got a little bit sort of like nerdy around that kind of thing. So I was was super into it. So it’s super fast.

And ⁓ there are some things that you kind of like needs that sort of is better for information to be presented in the UI. But for the majority of things that you’re doing, like setting prerequisites and that kind of stuff, checking connectivity, ⁓ updating courses. I mean, it’s just way faster to do it using the command line. So.

Jason (21:48) You

were skeptical of some things and I would say, can I replace this and take this away? And you’re like, whoa. You’re like, really nervous about me taking away some stuff. Cause like, I want to deprecate this because this is just old, it uses a bunch of old stuff. We got to get rid of this and we’ll give you a tool. And you’re just like, I don’t know.

Alex (22:10) Yeah, maybe there were some things that I was a bit like, wait, what’s he doing? I wake up one day, so the thing I need is just gone, you know.

Jason (22:19) happened

a few times where I’ve changed some stuff and you’re like, what happened to my thing? mean, this is, you know, you had a reason to be nervous whenever I was talking about, you know, deleting stuff, you know, but, um, but it was always funny. would, but, you know, I’d be like, Justin, could we just build like a, you know, tools? And of course it’s just way faster to, to, build them, to maintain them, give you all the cause cause a lot of times these tools have lots of

specialized parameters options, guess, right? A bunch of options. And that stuff is harder to implement well in a UI without requiring just a lot of UI work. A lot of, it’s just, it’s often not as it sounds these, ⁓ you know, just, it’s just always more complicated than

Alex (23:07) And as the sole UI developer in the company, I would much rather you’re working on the UI stuff for the customers than for me. like, yeah, I’m happy with command line tools. I don’t want to be hassling you with like, there’s a bug in the UI that’s preventing me from doing this, which probably takes like two hours due to, I mean, it depends, but you know. Yeah, I think it’s worth that.

Justin (23:29) developers

you need to allocate their time as efficiently as possible.

Alex (23:33) Because

there have been people on Twitter. I’ve talked about the Commanditives. Why do you have a command line tool? Wouldn’t it just be better to do a UI? It’s like, no, it wouldn’t. It really wouldn’t. This is just so much faster. And it helps us, helps the product move forward so much quicker than it would otherwise. So yeah, I’m happy with my command line tools. I’ve got no need to change anything. The only thing I probably would say, an analytics dashboard would be kind of helpful. But we have analytics tools. So it’s not like I’m.

completely in the dark or anything like that.

Jason (24:06) Yeah, no, there’s lot to be desired in terms of collecting analytics and making, mean, ⁓ Justin has some reports that he runs, more with user behavior and churn and this and that, but there’s a lot of the stuff that we should just build out in some tool, probably start with tools and then build some dashboards. yeah, mean, well, having the tools was hugely effective. Also it was great because you could complain to me and I’m like,

Go try to jump. I don’t want to, you’re like.

Justin (24:40) Yeah, freeze up a lot of your bandwidth.

Jason (24:42) Wrong

department. Yeah, was, you know, speaking of it was sort of funny, like when I think back in the earlier days, ⁓ you know, initially, you know, just his background was, you he was you did machine learning quantitative stuff purely. And a lot of it was sort of like, you know, when when when he worked at that analytics company, there were engineering teams. And so

He would just work with models and data and they would build out the infrastructure. And so he would do stuff in like Jupyter notebooks and stuff and everything was Python. that’s how kind of we started. He would write everything in Python and I would actually convert it. I’d rewrite it in JavaScript because we were in we were a node on the back end. Right. And eventually I’m like, all right, this is bullshit. You got to you got to write this in JavaScript. You’re like, I was like, I’m like, just.

And he was like, okay, all right, I’ll start doing it. And so he started doing that. And then he would push the code, know, commit it. And he’d be like, all right, it’s ready to deploy. And after a couple of months of that, was like, all right, this is bullshit. I like, you know what, here, you’re gonna, because we had it running on a whole separate server, right? This isn’t the main app server, a whole separate, this is our, had our first sort of engine, you know, whatever server. And I was like, all right, Justin, look, you’re gonna.

you’re going to deploy this code yourself and you’re like.

Justin (26:11) Yeah, yeah, I came in with very, little software engineering background. mean, like I could basically, I was like a ⁓ just mathy code stuff, research code, notebooks, little scripts, like leak code type of problems. Yeah, I had never. No UI. Yeah, no. ⁓

Jason (26:32) Do I work?

Justin (26:37) But yes, there was definitely a spin-up period. ⁓ But it was was one of those things where it’s like, I guess back to what we’re talking about at the beginning of the show is like, okay, I had a strength in math decoding, but until then I had kind of just like leaned into that so much that I never developed the rest of my coding abilities. And that was a severe weakness. But the silver lining of these severe weaknesses is once you…

once you shore them up, then you become so much more capable on so many more fronts. And so, yeah, I mean, it wasn’t fun realizing just how lacking I was on the software engineering side of things as opposed to the quant. But once I just dug in and leveled it up, suddenly there’s all the sorts of things that became available to me that I was able to start working on.

more work, more kinds of work I was able to start doing, taking off your plate that I wouldn’t have anticipated. And ⁓ yeah, it turned out really, well.

Jason (27:40) If you’re, if you’re a call. the, so the, this will happen was when I first asked if you wanted to work with me on this project and you were like, hell yeah. Right. And, we would pair code at my desk. just would sit here. I would sit here and we would pair code for hours. And Justin, you are working full time at the time. think you were still working at that private school and, um, you would come home, you would be.

here at like 3.30 and we’d work together until dinner time. We’d eat dinner and then you and I would sometimes work through the evening unless you didn’t have something scheduled with Sanjana or something. But we would pair code for hours and hours and hours. And ⁓ I would kind of just talk about the philosophy of software development. I’d write, as I’d write, short functions, shorten them and then shorten them more, simple classes, really, really pay attention to

how you name things, because if you can’t name something, you’re trying to do too much, or your thinking isn’t clear enough. It really helps if you want self-describing code, if you have to write a lot of comments, that’s usually a danger sign. So I would just go through all this just the heuristics of software development, not just write coding, but how do you write larger systems like short files, sort function.

cohesive, make things object oriented, you know, whatever. And it’s all about the art of managing complexity. And of course, were picking that up quickly because you’re initially just writing like, code like most quants do you would X and Y and whatever.

Justin (29:19) Yep.

look back at old code like that and just cringe internally. It’s so, it’s so bad, but yeah. I mean.

Jason (29:27) You know, it’s just what you do. mean, I mean, because your code is never more than I mean, as you all these grad students, undergrads and grad students in math and physics and, you know, who are computer who are writing stuff for classes, their stuff is never more than a few hundred lines long, which case, complexity is not the problem. Managing complexity is not the problem. Right. You know, having somebody else have to look at your code when a bug happens at midnight and they’re like, what is is not your problem?

So didn’t even, there’s no problems that we’re solving. The problem is like, how do I get this algorithm to do these things I need to do because that’s the assignment or that’s my research and I gotta get this stuff done. when you’re writing software and you have these very complex system, I have millions of lines of code and all this stuff, you have to think about everything completely differently. So anyway, we did that for, I don’t know how long, a couple months or something like that. But as of time, I’d be like,

You’d be like, well, I can just kind of work on this part and I could, and you go sit over there in the, in the living room and you’d kind of work on the quanta code. And you, you, at that time you’re doing it Python mostly still. And I’d be here working and you’re like, right, well, I got it. And then I’d convert it. And then eventually it was like, right. And then over time you would write bigger and bigger, more and more code that was just solid. And that was just me just kind of like, okay, merge then. And then I was like, okay.

And then, you would need code. You’re like, can you deploy this code? And I’m like, I’m at dinner. I can’t deploy it now. It’s like, we’re gonna put this fix in. There’s kids that have it. I’m like, all right, like this is bullshit. I, you need to be able to deploy stuff. And I remember you were just like, like I said earlier, you were just like, And I’m like, look, just, you know, SSH in the server, then I’d give you all the stuff and then you started doing it. But what’s really funny, because the learning curve happens is then you deploy something and it blows up.

Justin (31:20) Yeah.

Jason (31:21) All of sudden,

all these errors start happening. E-mails are coming. And then that was a whole nother learning curve of how to write bulletproof code. Because there’s only so many times of stuff blowing up that just is so anxiety inducing. It’s so stressful. It’s such a horrible experience. All these emails, nobody’s got tasks, this thing’s wrong. know, and so then you start going through the process of learning how to write code that won’t, that’s bulletproof, that’s

that won’t fail over, that won’t fall down, that’s logging everything, like what happened? You know, I don’t know what happened. We’re not logging stuff, or you know, how to alert us. I don’t know, so I talk a little bit about that, because I thought it was really interesting. I mean, the biggest growth happens when somebody says, you take care of it, and you’re like, oh, shit, you know.

Justin (32:11) Yeah,

yeah, yeah, that was that was, ⁓ yeah, pretty much right. So it’s like, I kind of felt like you just gave me this this code baby, this this robot to take care of. Like, don’t kill it. Like, I trust you. But yeah, it was very much. ⁓ I mean, it’s just stuff that you can’t. I mean, okay, I did not study computer science in college. ⁓ So I didn’t really have any sort of background in this. I was just

Jason (32:23) You’re good,

Justin (32:39) learning the hard way, right? School of hard knocks. just, all right, crap code blew up. Well, I guess they don’t really teach that a whole lot in computer science programs either. I guess everybody learns this way.

Jason (32:53) developer, engine engineer, know, production code is where you learn it.

Justin (32:57) Yeah, yeah, I remember. Yeah, it’s just various things. It’s like at first there’s just unknown unknowns that you just don’t like, you don’t realize that like there’s things like cascading failures that like, like one thing fails, it causes another thing to fail. You don’t realize the magnitude of a screw up that can happen just from one small thing. And so you think like, well, I’ll just make things not break internally.

Right. ⁓ that seems like a good idea. And that’s the first step, but then there’s another step like, okay, well, what if something does break? What happens then? We just need to, we need to limit the breaking and its scope, not to have it like cascade out to other things. Cause, cause just something’s going to break at some point, even if you’re as careful as possible. Like that’s just, you’re going to deploy a typo or your, your tests are not going to catch some case.

or somebody is going to run into some scenario that you just never thought would have ever happened, or somebody’s data is going to be corrupted for whatever reason. And then you got to make sure that that doesn’t screw up with other people. it’s, yeah, there’s just lots of, I guess, more and more advanced sort of error handling and writing this kind of robust code.

Jason (34:14) That was what you had this the selfie the auto what is known as autonomic code with code that can self heal right? So that you started writing some audit autonomic code in the model so that if something got screwed up so if it’s Alex had made a change and there was a problem that the validation then check that it would blow up for everybody because because that happened that would just like ruin your your night right? Yeah.

And because Alex is doing this in the middle of the night and all of a sudden, bam, four in the morning and you’re just like, what is happening? all these emails from people and nobody has tasks and you’re just like,

Justin (34:47) ⁓ Yeah. Yeah. Right. So it’s like, don’t, don’t let data into, don’t let the system be adjusted by the command line tools, unless you validate beforehand that nothing’s going to screw up and not just like theoretical validations, but like actually run the, have the model go and build its knowledge graph and everything to the point where like you are a hundred percent confident, like it’s guaranteed to be able to select tasks, run that same exact code every time you try to make a change to the knowledge graph.

Because that’s the standard that the change has to meet in order to avoid these disasters.

Jason (35:25) That was when we would joke. We always started saying, like, if you haven’t tested it, it’s I mean, that’s just it is. It’s like so often. And that’s the case. You might as well just assume it’s broken. And but the other thing that was really interesting around this time and ⁓ is when we initially had the first real version of the knowledge graph working. You know, this is back twenty twenty one or something. I don’t know. And we would you would calculate you would have to update

a student’s tasks based on their updated knowledge profile. And they would go through all the answers that they had given. would go through an entire answer stream and project that onto a knowledge graph and percolate all the changes up and down to find out what their knowledge data is and then figure out what are the optimal set of tasks for do and select the tasks and stuff. And I’m rerunning that and I’m like, well, how long does it take to run it? And it was like a student who wasn’t super advanced, but had done stuff. You’re like, about like a minute, and half. I was like,

That’s never gonna work. You’re like, what do you mean? I’m like, like a second or you’re like, what? I’m like,

Justin (36:27) Yeah.

Yeah, this was my first intro to real time because like, mean, previously I’d been like hooding up like research kind of stuff, like computational neuroscience or whatever. It’s just like, you just run a simulation. Like, what do mean? It runs in a minute. That’s fast. Like I can just, no, that’s not fast. That’s not fast when you have, when this thing needs to be run over and over again. And especially like when it’s going to have to scale with the number of tasks that are completed on the system.

Yeah, that was a lot. That was a lot of good learning and just how to leverage caching to speed things up. There’s all these sort tricks.

Jason (37:10) Yeah, was the thing I kept that I would say whatever telling in there later is like you can’t make code run faster. You could just make it the computer do less. And how do you make it do less? You don’t make it do the same thing over and over again. And how do you keep for doing the same thing over again? It’s like you identify stuff that it can reuse from what’s done before, hence caching. what information do we have? So what can we calculate, precalculate and save in a certain state that we can just pull from? And so there was a lot of work done from that that got it down.

100x faster. ⁓

Justin (37:41) Yeah, yeah, yeah. was one of those things where it’s like, I mean, yeah, coming from ⁓ a mathy quant-like background, like caching, like you don’t really think of caching like just as internal to an algorithm to make the algorithm run fast enough, but there’s like this, that’s just on a small scale, is large scale caching of all these objects that you need that are feeding into various things. And it was like previously, just, for whatever reason, I didn’t realize like how

broadened scope, you can push this idea of caching, which I mean, it’s something that we realize over and over in our work here, right? Which is like, if there’s a solution that solves one problem, like you can probably push it a lot further than you’re thinking. You can lean into it. I mean, just, I mean, for instance, just the idea of like spaced review in learning, just not just tech, but like everywhere, space review. you…

Like, okay, good, let’s, let’s prevent students from forgetting the material. But also if you really lean into that, you can actually make it super efficient so that they barely have to review stuff. They’re getting a lot of review credit implicitly. There’s all these like little things that, ⁓ yeah, you can just push way, way further than you would expect.

Jason (38:57) You know, was was, I mean, cause when you first started wearing it, I think we talked before, you were like 22-ish. This was 2000, summer of 19, 2019.

Justin (39:07) Yeah, I was 22, turning 23 in August, but yeah, was 22.

Jason (39:13) Right. you’re young, you had never written production. I mean, you had had lot of experience. You worked all through your undergrad years, just three years in analytics company. So you were really advanced for your age in machine learning and sort of commercial grade analytical ⁓ type of building analytical solutions, but not on the engineering production, real time, all that kind of stuff. And then ⁓ just proper software development. ⁓

you techniques. So but, know, of course, you leveled up quickly. One, because, you know, I could sort of give you lots of shortcuts. Hey, just here’s how you want to think about it. And so you’re like, all right, fine. You would you were highly coachable, extremely coachable. You would just I tell you one time and you’re like, OK. And then so you leveled up. But second, I was like, all right, you take it like yours now, because I can’t I have too much. so then you’re.

And then of course, it was sort of, I always sort of laugh. mean, it’s like when things did blow up, I’d be like, that sucks. But I’m like, well.

Justin (40:15) This is a teaching. Yeah, exactly.

Jason (40:18) I can tell you’re a frazzler. can see you’re like, oh.

Justin (40:21) Yeah, I remember, I remember that I think the first time I accidentally blew up the task processor, I was like, I nearly puked in the in the toilet. Like, I was just at my place. So you didn’t see all this happening. But I was like, for a good minute, I was like, just feeling like I was about to pass out or something. But then it was was very stressful. But it was, but that was, mean, like we always talk about, it’s like that emotional scar tissue just guides you in the future. Like, okay, we’re not gonna do that again. We’re, we’re gonna

Every time I’m writing code now, I’m thinking like, how can I not be in that situation? And then I haven’t been in the situation since. ⁓

Jason (41:00) Is

that like I had to be upset about it? I’m just like, that sucks. I I just get there and go, I’d be like, right. Well, how’s it going? You’re like, I’m thinking I just like, all right. Well, you know, and like he’s is because I, you know, I’ve done it, too. I’ve blown stuff up my share at times and it sucks. It sucks. But it definitely teaches you to have respect for the process and get you to do all the right things. It’s just like, I don’t want to have that experience again. It’s just it’s terrible.

terrible. You things blow up, people are upset, everybody’s confused, the customer support emails are coming in, in a flood, you know, nobody can log in or do, you know, and of course now we’re at a scale where it’s like, it would be just like, bam. I mean, that’s why it’s good to smart, to start small and then like grow organically. Cause then you’re, you’re the, these skills and tools and processes and everything can

to evolve over time so that when you do have a larger scale, it’s not like, no, what do we do now? It’s like, it’s just 1 % bigger than it was. But then what does happen, and this happened with the database server stuff, things will just suddenly hit a point and then it just goes like this. The pressure on the database, that was a big problem like two or three months ago.

Justin (42:27) Yeah, right, right. I think it kind of comes down to, again, like the cascading failure thing, where it’s like, get in a regime where one thing goes wrong and your system is kind of in this weird state where it’s being pushed up against some limit. And then suddenly, like these failures can spread all over. ⁓ Yeah, right. I remember, yeah, I think it was last summer, actually, right, when the database hit capacity and…

But it was like, was kind of getting pinned up to this like 99 % utilization. And then, and then it was like people’s tasks started to become slow or like the dashboard loading page. Cause like the, guess there was a, there was a particular query, but I was pulling in all the data for the leaderboard and everyone’s page that was kind of running slow. Why was it running slow? was running slow because the database was pinned. And so this query wasn’t running as fast as usual.

And so because the query wasn’t running as fast as usual, that means everybody is just kind of hanging on this dashboard page. What do people do when they’re just hanging on the page? Will they go refresh the page? Now that’s running more and more queries. Now we have like, it’s just this huge explosion of like, just, yeah, bad. So, right. So like learning to just keep everything in a, in a stable state and yeah.

Jason (43:48) Because it went, it was like a 20%, 10%, 15%, 20%, and it started getting up. And then it went to like 80%. And then within a week, it was at 99.8%. Like it just was like, you’re like, okay, is this a little bit, maybe we look at some problematic queries or are we missing an index or two or on some tables? I what’s going on? And it was just, the growth, you just hit these inflection points with growth. So you have to be careful about keeping your margins. You need to be operating with some…

some headroom otherwise, or have infrastructure that can auto expand in the way that you want it to, which is sometimes harder than you think. And we’re more expensive to get all that stuff in place unless you’re some massive company that has a lot of money and you can just do all this. So Alex, are there any more topics that you, because you hit anything you want to talk about? I mean, we’ve been going for over two and a half hours, so we don’t have to.

Alex (44:48) Well, there’s a couple of things I thought maybe we could talk about.

one thing I wanted to ⁓ discuss was how we deal with some of the smaller problems that occur, that we hear about at Math Academy from students. So I was listening to a video of Jeff Bezos a couple of weeks ago, I think it was. And he was talking about some of the processes that take place at Amazon. And of course, you’ve got the big, exciting projects.

the new data centers, all the cool UI stuff they’re doing, new products, all this kind of thing. But he was talking about, okay, so those are great, but what we have at Amazon is a paper cuts team. So when a customer has an annoying, unpleasant, but not big scale, unpleasant experience, that’s a problem for us. We need to kind of root those out and fix those kinds of problems. And I thought to myself, that’s a really interesting idea.

the idea of having like a paper cuts team because we hear about things like that all the time, you know, ⁓ I mentioned to I was just mentioned to Justin that one of our customers, he’s got a nine year old student on the system. And it’s gotten to the point where this nine year olds ⁓ math ability is kind of outpaced his reading ability. So he’s actually reading all this stuff, which is really aimed for like high school reading ability students. And he’s nine years old.

And he’s running into problems when not only are the, the, is some of the texts, maybe just too much for him at this, at that age group, but also in certain concepts he just has no idea about. Like for example, he was doing our, we’ve got a topic in algebra one called mixture problems. And he’s like, one of the issues, he doesn’t really know what a mixture is. He hasn’t had the science background to really appreciate what is a mixture. When you say it’s a mixture of, you know, water and salt, well, what does that?

What does that mean? 20 % sort what? You know, so it’s a hunt. never I never thought about it. I never thought that that would be a problem. So that’s a paper cut. The kids coming up against this topic called mixture problems. No idea what a mixture is too much text. Okay, that’s that’s a paper cut. They managed to get through the lesson. But you know, obviously, I’m sure you might doesn’t mind mentioning his name Rahim who’s who’s kid is he sat next to him and got them through the lesson. But it’s like I need I need someone in the team.

on the content team who could deal with paper cuts. Whenever I hear about paper cut, can say, right, this customer has had this problem, here’s a clone of the topic, do whatever you want, but fix that paper cut. Make sure that paper cut doesn’t happen anymore. When you’ve defined what it is, we’re fixing problems with existing content all the time. When you define what it is, it’s like this is a paper cut and we need a team to deal with those paper cuts.

so that they go away, kind of ⁓ draws a box around, is this thing, is our problem, which we’ve kind of shaped, and you can now give that to someone whose responsibility is to make sure that that is being dealt with. as we grow and get more more reports from people about these kind of things, we need to have a proper process at dealing with them. I think we’re great when it comes to new content, rolling out new courses and stuff like that. It’s all very efficient.

We don’t have too many teething problems with new courses, which is great. That’s brilliant. That’s like the big infrastructure products at Amazon. You cannot neglect those, those little issues, little annoyances.

Jason (48:20) years to refine this process, right? And a lot of introspection and sort of like troubleshooting the inefficiencies to get it to be as efficient as it has over the years. mean, obviously, in addition to tooling and things, but just your processes, right?

Alex (48:22) Yes.

Yes. And that’s been our focus, know, sort of growing the content base, growing the knowledge graph and obviously taking care of the really big problem. If like 80 % students are failing on a lesson, that’s not a paper cut. is huge problem. So that requires a

Jason (48:50) Even

if something falls below 20%, if 20 % fail, if it below the 80%, we’re like, well, what’s going on?

Alex (49:00) Exactly. So that is is kind of paper cut territory. So it’s like, and there’s one one ⁓ content developer on the team, Andrea, who’s been with us for seven years, she’s always really impressed with her work. And it’s just like, hey, now it’s time, I think, to get Andrea involved in, you know, perhaps some of this, she knows that she knows the content inside out, she knows our standards inside out, she just sort of diligently done her role.

for a long time. like, okay, we need to solve this problem. I think this is the right person to do it. So hopefully she’ll be the person kind of like heading that up. ⁓ It’s just great. Also, yeah, it’s just great to kind of like name it, name the problem, give it a catchy name that no one’s gonna pick. Okay, it’s not that catchy, but give it a name that you can point to every time this kind of thing happens. So you can have a person that’s responsible for that.

Jason (49:35) Gone, sorry.

Alex (49:54) And then you can make progress. Because I think that’s one area where we need to improve, I think, when people have those smaller problems.

Jason (50:02) just getting on them and dealing with them quickly in a timely fashion.

Alex (50:06) Dealing with it quickly, coming up with solution quickly, shipping it quickly, doing the knife switch stuff quickly. That’s something we, I’m really keen ⁓ to get that process as good as our regular content development process is.

Jason (50:22) Well, it’s sort of similar with what happened with Justin and I. Sometimes you just got to carve stuff out and hand it to someone. like, just can’t, everything can’t go through me. Right. And like so much goes through you for reasons of just quality control and just because you have so much more knowledge about everything that, but it is a bottleneck. And so if there are things it’s like, okay, like you’re identifying, there are things that do not have to come through me.

that need to be done. Cause if it just comes through you and you’re like, well, I’m dealing with this course, I’m dealing with that course. then, you know, it’s like, okay, and it’s just like, it’s the 17th thing on your list. And people are like, well, hey, we reported this a week ago or three weeks ago. Why isn’t this dealt with? And you’re just like, well, on top of everything else, Jason called me up yesterday and was like, Hey, I got this idea, you know? And so on top of everything else. you’re, so you, right. So having, having someone, you can just say, look, just all these problems, I’m just going to send it to the, you know, captain of the

paper cuts team and she’s just gonna make it go away. You she might ask me a question, but she will just make it go away, right? I mean, that’s the ideal is that she has the skill, the knowledge, the conscientiousness and everything to just make these problems go away. So would you build a team for it, Ryan? Would you give her some people over time, have her deal with it herself at first and then add a couple of people to it or what are your thoughts?

Alex (51:45) Just her to begin with. So she would do essentially what like, like Yuri and Josh do on the team. It’s like they present me with like their proposed solution to whatever happens to be we need a topic on X or topic Y needs to be improved. She does everything that needs to happen. And then she shows it to me and hopefully I go, yep, that was great. It was great. It was great. And then we ship it to what we call the production team. They do the mass producing of the questions that we need. And then from there, it’s quite a

a streamlined process to get it into the actual system itself. So there probably will be some back and forth in the beginning. But once she’s competent, once she’s really competent and knows exactly what she’s doing, then that’s the point at which we can maybe have two people doing that. So I don’t imagine it being a very, very big team, like one or two people maximum, ⁓ at least for the foreseeable future. But one thing I’ve realized over the years is that for this kind of stuff, for content development, because

We’ve got so much and there’s so many conventions and standards and things you have to bear in mind for creating content scaffold. There’s just so much to bear in mind. You can’t train four or five people at the same time. It’s it’s it’s impossible. One is pretty much spending a lot of time with one person.

Jason (53:02) Does she have, is it one of her strengths that she has a broad enough knowledge that she can handle something on different equations and problem statistics and, you know, different, you know, whatever abstract algebra, I she can, you know, span these areas like you and Josh and Yuri. Is she a teacher?

Alex (53:21) She’s a mathematician and she’s extremely competent. mean, she’s like I said, the reason I’m sort of considering her for this particular, well, we’ve had the conversation is because whenever there’s been issues with a topic, she’ll often say like, oh, here’s what I would do. Clone the question, tweak the explanation. Here’s what I would do. I know there’s ever been an instance where I’m like, sure. It’s always been, yep, she’s nailed it. So that’s, so she’s.

You know, ⁓ so yeah, I think that she’s I think she could do a really incredible job with that. because she’s been with us seven years, she’s seen the full breadth of the curriculum. She’s seen the kind of differential equations we’ve got and the early years stuff. And yeah, so I think she’ll she’ll do an amazing job is what I’m hoping. I mean, there’ll be some training involved, of course. But yeah, I’ve got no.

Jason (54:15) That’s great. Well, that sounds great. We need one for technology too. We need like a paper.

Alex (54:21) On that point, so I actually announced ⁓ I thought I’d test the water with a whole paper cuts idea. So just on X a few hours ago, I said, yeah, I’m thinking about setting up a paper cuts team for math academy. And we need to iron out all these little problems that people have. So of course, people have responded. And I’m pleased to let you know that every single paper cut so far, what will they consider to be paper cuts or UI things? ⁓

Justin (54:48) Me! Not even a single cut.

Jason (54:50) We’ve all agreed it is a problem and it’s you.

Alex (54:56) Yeah, so…

Jason (54:58) Well, there’s a lot of surface area on it, so it’s not surprising. That’s true. Yeah. Yeah. Well, when Colby…

He’ll probably put in some work over Christmas. you know, my son, Colby, who’s a senior now in college, he’s studying math and computer science. So if he he graduates in June, then he’ll be full time Math Academy, because that’s what he said he wants to do. And so and he’s he’s full stack. He’s really good. He’s got he’s got a ton of features that actually I got to roll out. He’s like, Dad, you still not rolled that stuff out? Come on. But.

He may go back, he has one more, I think he has to do one more quarter, he has like two courses to complete his master’s in computer science. And he’s kind of like, that doesn’t matter, nobody cares. And I’m just like, well, it’s, you know, one quarter, like might as well, you know, might as well get your master’s degree. it’s very, the worst case will have him next year, but might have him sooner. I mean, well, he’s gonna have him the summer full time. But he’s…

Yeah, we’ll have to look. mean, we may expand the team a little bit. I’m just always very, a lot of times it’s just, I found you expand the team and with really hard. mean,

Justin (56:17) really hard.

It’s a harder than I thought it was gonna be. I knew it was gonna be hard, it’s just, there’s so many challenges to be overcome. ⁓ yeah, the fact that like Colby, we already know he’s got like serious development chops, that he’s really solid and that like you and him work well together. I mean, it’s so hard to find ⁓ other people who are not only capable, but also like,

work well together and also have the background knowledge on math academy, right? That’s another big thing is like, you can’t just like pull somebody off the street, pull a great coder off the street and have them like work on the math academy system. Well, there’s a lot of, there’s a lot of like domain knowledge about how this, not just like at a software level, how the system works, but like at a pedagogical level and this being in the mind frame of a student or a parent.

Like how does you’ve talked about sometimes about all the, these decisions that go into the UI, right? It’s like, well, it looks, it looks easy because like at the end result, like it’s, it’s supposed to look simple to people. That’s the hard part is making it look easy.

Jason (57:33) Making stuff look easy is the hard part, right? That’s right. That’s a very good point. Yeah, for sure. So we’ll see. I would love a paper cuts team for the UI and our app just in general, because there are lots of things that just have pushed off for the time being because we have so many big things that we’re trying to finish up and get out the door. So speaking of getting things out the door, let’s finish with one more thing. We can cover this more in the future, but…

The upcoming, the next course to be released is our differential equations course, which is well underway. And why don’t you just, let’s just talk a little bit about where we’re at, what it is. mean, whatever, however you want to describe it.

Alex (58:19) So, yeah, so we’re well underway with the differential equations, of course. I have kind of gone public and said that I’m hoping to release it before Christmas, because last year, we managed to release like last year, probability statistics on Christmas Eve. And that was what I wanted to do is like, that’d be a great thing to be able to kind of maintain. ⁓ I’m slightly regretting actually saying that now, although I’m still aiming for it. The thing I’ve realized,

is that a lot of these stuff on, so we’ve actually had quite a lot of differential equations stuff like in the background. So if you go through the Math Academy system and you complete all of our high level courses like linear algebra, ⁓ methods of proof, ⁓ multivariable calculus, discrete math, and probability statistics, if you finish those, you start getting like what we call Easter egg topics, which are topics in the system, but they’re kind of like not in any particular course, which is available to the public.

And the reason those topics exist is to help out with the high school program. Years ago, they had all these kids finishing calculus in eighth grade or whatever. And so we need stuff for them to do. So we went in and we create a whole bunch of abstract algebra topics and differential equations. And obviously, eventually finished up all the multivariable calculus and linear algebra. eventually got them to piece into these actual proper courses.

But I’m looking back at some of this original content as there’s so much we’ve learned about how content should be created that we didn’t really know back then. So just to give you one example. So we got a probability and statistics course. within that topic, I think the course has gone down really well. But there’s one or two topics on moment generating functions where

Basically, the way the questions have been designed is very easy for the student to kind of like infer what the correct answer is going to be because it is like there’s this particular pattern that makes it very, very easy to go, yeah, that’s the correct answer. That’s the correct. See one or two and then they can just even if they don’t know what the hell is going on, they can just kind of get through it. And even though even the most well-intentioned students who wants to know it, they might be tired, whatever, and they’re just clicking and they’re spotting the pattern and just kind of getting through. And a few people complained about this.

So yeah, this is really silly, blah, blah. So we’ve got a particular topic. We’ve got a big area of the differential equations, of course, is going to be Laplace transforms. Now Laplace transforms are basically the same as moment generating functions in a slightly different context. And it’s like, OK, I’ve got to make sure we don’t make that mistake again. So I’ve gone back. Now Laplace transform is actually one of these Easter egg topics. So I’ve gone back and looked at it gone, oh, yeah, of course, this was created five, six years ago.

Of course, it’s got the same problem as that moment generating function topic. what I need to so basically, those topics almost have to be completely rewritten. What we’re going to do this time is rather than just multiple choice whatever, almost use dynamic select questions to walk the student through the calculation. So they really got to think at each step. ⁓ So.

Yeah, so there’s quite a lot of work to do in terms of fixing prior, like not it was exactly, but, know, to improve the content based on what we know is good practice right now. Of course, when we did the high school program stuff, you know, there were certain topics that we just were, I guess, were missed. Shifting themes and things like that. So you’re just going and backfilling a whole bunch of a whole bunch of stuff. So.

As a few other areas as well, one of the earlier topics is going to be first order differential equations. ⁓ What I’ve noticed, a lot of resources actually kind of dive into the method of integrating factors straight away, which you can understand why they do it, but actually they should go back and sort of. ⁓

do linearity first, really explore it, because there’s some missing scaffold in there which could be exploited. But to cut a long story short, there’s a lot of backtracking and putting things as they should be to create a commercially viable course. So Christmas is definitely possible, but it’s a lot of work.

Jason (1:02:45) So like one thing that I always, you know, when I would say, Alex, you know, this is what I, what I think we need out of this course is it needs to be at the level of what you would see at Harvard or MIT or Stanford or whatever. Right. So you don’t want someone to say, well, you know, that thing, a math academy, mean, you know, it’s, not bad. mean, it’s, it’s kind of cute, but I mean, that’s not what we do it, you know, at Berkeley or Harvard or something. Right.

You want to be like, yeah, man, like it’s legit. Right. And so, you know, one thing that I asked you to do when I, first created our, mean, we created our very first, ⁓ university level courses, linear algebra and multivariable is like, go look, what can you find? What are they doing? If you can see any published final exams or syllabi from these elite institutions, along with

the most popular and most well-respected textbooks that are used at those institutions, what are they covering? What’s the superset of the commonality? Justin would always point this out. There’s always some really weird idiosyncratic topic that some professor covers because maybe it’s related to their research and they just find it neat, but nobody else would talk about it and everything is weird.

If you shave those off and you’re like, okay, they covered this at MIT and this account tech and this is Stanford and this is Chicago and this at Duke and this, you know, and you go all that and you go, okay, what is that? Because you don’t want to go, well, I don’t cover this. I’m going to cover this at MIT. It’s like, well, they didn’t cover that at Stanford and they didn’t cover that at, you know, at Oxford, but you look, you look pretty broad, far and wide to just make sure that we’re covering all this stuff. Right. I mean, tell me, just talk a little bit about that.

Yeah

Alex (1:04:41) Yeah, in terms of what differential equations course should look like, there does seem to be a sort of general consensus among the top universities, like US universities. So we sampled, I think, like 8 to 10 of the top universities in the US and came up with a list of topics which they all seem to cover, plus a handful which some of cover, some of them don’t.

And it was from there that we really came up with our first course outline. Now, we are somewhat constrained in the sense that ⁓ it needs to be a semester-long course. So maximum 200 topics for a course like this. So that is of a constraint. But I think that the differential equations course we have planned is

is going to cover everything that you see at most of these universities. mean, you always get people online and ask, hey, you’re to have PDEs, you’re going have signal processing and this and that. Well, they’re typically covered in later courses. We’re just doing an extremely comprehensive differential equations course, but it is an introduction to differential equations. It’s not going to be pulling in things that you wouldn’t normally expect to see.

Justin (1:06:03) Yeah, it’s the same kind of scope creep that we saw in machine learning or even in the linear algebra, right? Like people often confuse like Axler’s linear algebra done right. No, that’s a second linear algebra course. You wouldn’t see that in a first semester linear algebra course. So it’s kind of the same situation with differential equations, it sounds like, where we have to draw the line somewhere, right? We can’t just have it expand out indefinitely until you’re doing like research on Navier-Stokes. That’s your final topic. Like, no, we need to…

Keep the scope down.

Jason (1:06:34) Yeah, I mean, I think it’s important as we’ve talked about this for years is to have apples to apples comparisons with what we’re doing and what is what is done at a comparable level. So it’s like our algebra one course can be it has it’s a year long. It’s a nine month course.

Right? Because that’s what’s done in schools. When you think of algebra, what is algebra one? Now we could be more comprehensive because you only look at different textbooks and they go a little deeper on this or little deeper on that. And because we have extremely efficient pedagogy and learning algorithms that we can do more, but nevertheless, it still has to be a year long course. Now for university courses, they are a semester or quarter long course. for people who aren’t familiar in the US, of course,

probably three quarters, maybe more, maybe 75, 80 % of universities run a semester system. So the year is divided into two. So it be a half year course and you would take four to five courses. In some schools where I went to University Chicago, was quarter based. So you had a fall, winter and spring quarters. And so you would only take three or four courses. So you would do more in less.

time, if you were courses that would cover a little more ground. But regardless, they’re pretty much the same. It’s pretty much the same amount of material. So our university courses need to fit in that framework so they can compare and go, okay, as a semester quarter long course, math course, how does this stack up to what’s going on? If you do it like a year long course, are like, well, don’t, is this like a, and everybody gets confused.

And so we always say like, can reinvent a lot, but we do not exist in a vacuum. We exist in a world where things are done a certain way and people have certain understandings and expectations. And if you drift too far off that, then people just do not know how to think about what your product is or what you’re doing. And it just becomes so much friction in crossing that, that it’s just, it caused the product to fail. So we need to be, okay.

We go, ⁓ so it’s calc one. Calc one is a semester, quarter-long course. Calc two, I get it, linear algebra. So we have to kind of do that. So I think you’re right. Put a cap on it. It’s ordinary differential equations. It’s going to be a super set of us of stuff. So if you go to one of these courses, you won’t be able to go, we don’t cover that. It’s like, hell no, we do cover that, cover all that. But we’re not going to go crazy. But we can do the differential equation, partial differential equations.

course or something like that as a second course later. I I think we have our sort of goal is to get the lower courses first. So I think differential equations is sort of like a legitimate sophomore or junior level course, depending if you’re math, you might take that your sophomore year. mean, would take, you know, the standard is you take Calc one and Calc two as your freshman year. You would take linear algebra.

multivariable calculus your sophomore year, maybe differential equations, or maybe your different equation would be a junior year. I was pure math, so I didn’t really took different equation. Mine was real analysis, abstract algebra and stuff. but if we get this down, if we get differential equations down, I feel like we’ve covered most of the lower introductory level, sophomore, junior level stuff. Now pure math, of course, we would have to do real analysis.

you know I covered that. I mean Alex go on Alex I mean just go on.

Justin (1:10:20) Yeah, but yeah.

Oh, I was just gonna say, yeah, this is like the last course in the core engineering sequence for an undergrad. I think that’s like the special part about it, but yeah.

Jason (1:10:35) A lot more people need to know differential equations than they need to know real analysis.

Justin (1:10:40) Yes. Yes. ⁓

Jason (1:10:42) It’s a better or abstract algebra. But then after this, so the goal for differential equations is we’ll have it done by the new year. And then after that, it’s on to, I mean, obviously we’re gonna try and roll out machine learning, but that’s not really a content thing. That’s more a technology thing. At this point, the content is basically done. It’s abstract algebra. Alex, is that what you’re thinking?

Alex (1:11:08) Yeah, I think we can get the ball rolling on abstract algebra. we we’ve I mean, maybe this is a conversation for another day, but ⁓ upgrading the kind of like the proof infrastructure. ⁓ And probably rolling that out into something like methods of proof first, and then but working on abstract hours, because not all abstract algebra is going to be is going to be like require like freeform proofs, there’s still a lot of stuff we need to do on the content end. So ⁓ but yeah, broadly speaking, ⁓

get some proof infrastructure in place, then methods of proof, update that to have the free form proofs, then abstract algebra, and then rehearses.

Jason (1:11:45) Yeah, okay. yeah, let’s, we won’t go into detail on that, but so differential equations course, how are feeling about it? mean, is it, as I guess, of course.

Alex (1:11:56) Yeah, I feel good. it’s, it’s, yeah, I I think that we are kind of like,

I’m excited about it for sure. ⁓ You know, it’s because we’ve got all this existing content already and we’re of like filling in the gaps. It’s kind of like, it’s a little bit difficult to see the wood for the trees in some sense at the moment, but I think it’s going to be, I think it’s going to be a great course. And, and one thing is, as well, I was going to say is that there are opportunities to enhance this course. Like some of the, some of the, like the, numerical methods.

We’ve learned a lot about how to deal with highly numerical work, basically doing algorithms, by hands. Stuff like that. ⁓ In the calculus courses, we’ve got something called Euler’s methods, which took a few iterations to get right.

Jason (1:12:39) and things like that.

Alex (1:12:51) But now that we’ve got it right and the pass rates for that are really high, you just need to carry that same methodology through to the differential equations course. We know it should work. It’s the same thing, but it’s slightly more complicated. But then once we have the coding infrastructure in place, we can have coding courses. Students can complete the loop. like they do it by hand, they get proficient, and then we give them a cool coding project where they actually have to.

predict something about the weather using like the Run Jakarta method or something like that, which I think would be really cool and help to of square that off. ⁓

Justin (1:13:25) Yeah, that’d be awesome. Cause differential equations opens up so much surface area to cool modeling projects. mean, I remember back when I like was really getting interested in applied math. was the thing that did it for me. Differential equations, modeling everything, biology, weather, whatever. Yeah. Sure. We could get a ton of cool projects out of that.

Alex (1:13:44) Yes, there’s lot of there’s gonna be a lot of potential. Once the course is done, there’s a lot of potential there in like a V2 where we do do some more projects that involve coding and modeling and things. So yeah, that’s that’s gonna be really exciting.

Jason (1:13:57) ⁓

Yeah, I guess it’s like the coding thing is interesting because we could, there’s certain, we could have versions of courses that require coding or don’t, you know? ⁓ It’d be interesting to have like, you could do differential equations, you could do linear algebra, you could do, you know, all these things, multi-row, you could have a code which has a ton of coding projects with or without coding. it’s like linear algebra with coding or something.

Justin (1:14:23) Yeah, it’s like a topping, like a pizza topping. Yeah.

Jason (1:14:28) Yeah, whatever you’re not, you know, and so we were like, don’t I don’t know how to code on or out. But some people like, yeah, because that really will bring it to life. For a lot of students is like writing the code writing going to kind of talk about building the tools, you know, you you once you write the code to do stuff, it’s like you just kind of mentally, at least for me, I mentally own it, cognitively own it. My gap. I got it now. I totally get I can see the solution in my head now. But one thing I was just saying, the oilers method was

The Moller’s method topic, I kind of remember that. That was a real problem early days, right? We had a really low pass rate on that. that took some effort, didn’t it?

Alex (1:15:05) It

took some effort. Yeah, it stands out as being one of the topics I found that was quite hard to teach in like an online in our settings. We just didn’t I can’t remember. It’s quite a long time ago now, so I can’t quite remember the effort. But I remember we tried to teaching it, you know, as you might find in regular textbooks. I didn’t work exactly. So we iterated on it. Still didn’t work. I think it took sort of three or four goes. And in the end, we came up with like this really sort of sleek, simple notation. ⁓

a of like a tabular approach, which kind of avoids a lot of indexes. I think I call it index hell when there’s just unnecessary index, know, Y, I, J, all that that kind of stuff. You want to avoid all that as much as you possibly can, because it’s just it’s just cognitive load. Was that index mean? Was that index? This approach that with the approach we settled on avoids all of that stuff. And so we’re just going to carry that through to these other other methods.

⁓ There’s other things you can do to help build intuition. before we even start with Euler’s method, we get them to do like a tangent line approximation by hand. So they visualize, OK, this is what we’re actually doing.

Jason (1:16:15) This is what we’re

Alex (1:16:17) And then you formalize it in the form of Euler’s method. And then once you’ve got that down, it’s like, okay, well, now we can improve on his method because all his methods kind of sucks. ⁓

Jason (1:16:26) When I taught it at the BC calculus course, my kids, hate that. They hate it. They’re like, you know, because it makes a big table and it’s boring. It’s process that’s supposed to be done with a computer. really a human. Unless you’re there in the 1800s and you’re like, well, this is the best I can do.

Alex (1:16:37) Hahaha

But you really need to do those, as painful as they are, really need to do them so they get under your skin. when you actually come to code something like that, you know exactly what’s going on. It’s very easy to look at a formula, create like a Python script, which is like 30 lines long. It gives you an answer. It’s like, well, I don’t really know what’s going on here.

It’s exactly the same mentality we’ve got with this machine learning course. It’s like, we want people to do gradient descent by hand. So it really gets under their skin. So when they actually translate that into like a coding environment, they know exactly what’s going on, you know. ⁓

Jason (1:17:21) It’s

gonna get under their skin in more ways than one. I wonder if they’re like, ⁓ they do that? But then they will know it. It’s like, dude, like now you know it. I know you didn’t love it, but now you’ve mastered it. Intuition through repetition.

Alex (1:17:38) Yeah, I mean, it’s one of those things. mean, like, you know, when it comes to things like gradient descent, I mean, you hear we were talking earlier on in the in the show about, you know, people that sort of like that they kind of went the top start the top down method of becoming machine learning engineers and never really did the bottom up part. So they kind of vaguely know what gradient descent is, but not really. And it’s like, that’s not a good position to be in. If you really want to make kind of like cutting edge technology or pushing the boundaries on things.

So yes, it is a necessary evil to kind of go through those painful calculations by hand.

Justin (1:18:13) Gotta get inside the trade, as you say, Jason.

Jason (1:18:17) Right. It’s such a you’re in your, in your beans. That’s where it’s like you learn, you develop that intuition and then you can get, can, the abstractions stuff all fall into place. But if you try and skip, skip the concrete examples, skip the repetitions and go straight to the abstraction formulas, you’re just, ⁓ you just like cargo cult math, you know, you’re just parroting stuff. It’s you sound like you know what you’re talking about. You know, it’s like, okay, they’re like, they can’t really solve anything.

Justin (1:18:39) wordy.

Alex (1:18:48) nice thing about some of these numerical methods is that even the more complicated ones like RK4, it’s like they are actually extremely intuitive if you actually spend the time to get the notation right, to get the examples right. So yeah, they’ve got every opportunity for it to get under their skin to the point where they know it really well without necessarily, you know, completely hauling over the coals.

Jason (1:19:16) Yeah. Yeah. Well, I’m excited. Well, I think it’s going to be sure it’ll be top notch and I’m excited to see it as it come together. So, well, I guess we should probably wrap it up. This has been another long one. I mean, I, Justin, I was talking about, keep it an hour and a half and hour, hour and a half. And just as like, have you ever listened to yourself? It’s not even remotely realistic what you’re smoking. So, all right. So we’re, three hours. Okay. Alex, sorry. Uh, you know, I this is much longer than

you were probably expecting or.

Alex (1:19:48) Actually, I wasn’t I was kind of expecting it

Jason (1:19:52) I know you. This isn’t my first call with Jason. You’re like Crammy, I’ll maybe see you tomorrow. I don’t know.

Justin (1:19:53) was planning for it at this point.

It’s good that we cover a lot of bases.

Jason (1:20:04) All right, cool.

Yeah, yeah. All right, guys. So I’m going to let’s wrap it up. And that’s a show.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.