SPEAKER_01:  Pretty much all math learners are experiencing a lot of friction during the learning process, and there are ways to greatly reduce that friction by trying to develop more interest in the material, by doing review, by doing mastery-based learning, space repetition, all these things that kind of reduce friction in the learning process, developing a habit, and so people can accelerate their math abilities well, well, well beyond what is the status quo.
SPEAKER_04:  Scripting bits is brought to you by the following sponsors, MED protocol, maximize your eaf seeking value with MED-Eaf exclusively on any .irer, and conversable, execute any intent on any chain, coming soon to mantis.app, as m-e-n-t-i-s
SPEAKER_03:  .a-d-p, and fast-side labs, trust the smart contract-based sequencing from validators and apps.
SPEAKER_02:  GMGM everyone, my name's Digachi, the host of Scraving Bits, and for the first time ever on this podcast we're doing a repeat with Justin Skysack, and for those that have not listened to their last podcast, it was episode 102. Learning mathematics like an athlete, we're back with round two for the sprint, after the marathon, just for like a small little intro, for those that haven't listened to that episode, who are you and what do you do real quick?
SPEAKER_01:  Yeah, sure, happy to be on again, great conversation last time. My name's Justin Skysack, as you mentioned, I'm the Chief Quant Director of Analytics at mathacademy.com. I built all of the quantitative software that drives the math academy learning engine for an automated online learning system.
SPEAKER_02:  There you go, and probably won't get too deep into it as we spoke about on the last episode, but this is just a continuing one episode. So if you haven't listened to the last one, we'll slow that and come back, or you can just stay and skip the intro and all that other shit, but it's okay. Sorry, I haven't done this before, so we're gonna see how this goes. Let's start off with proof writing, because I'm interested on your platform, you have proof writing, and I've been reading it quite recently, how to prove it, a structured approach by Valman, I think his name is, and I see this after you finish, let's say, Foundations II, Foundations III, or even just starting, or like the university stuff, I think it's super important to learn proofs, because it seems like if you know how in school, they're like, show you're working out, I think proof writing is that, but in the formal sense. And you know, let's say you make a discovery, and you want to solidify this as a truth, then you've got to make a proof, little buzz. But like, it's like the mathematical version of programming, instead of creating a program to kind of prove it, maybe there's underlying edge cases in the program, you just haven't hit yet. It's like a solidified version where you just attack at all angles, and then it's like a program that doesn't run, I see, and you can actually learn a lot quicker. I'm not too sure how, because I haven't done it yet, but I have the sense that it's like
SPEAKER_01:  that. Yeah, you know, I would agree with kind of comparing it to sort of like a program. I often, when I would teach proofs in the past, I would all introduce them as a, it's like a program that runs in the mind of somebody who has acquired mathematical knowledge. And that actually makes it kind of difficult to debug one's, for an novice to debug one's proofs sometimes, because it's kind of like, you know, you write a program, get the computer to run it, and it just, it compiles the program, it spits out whatever errors you made. But when you are writing a proof, basically the compiler is like a more advanced mathematician, like a teacher. Oh yeah. You can read it. But yeah, if you're novice, it can be a little difficult to kind of compile your own proof, because yeah, you need to someone with more advanced math knowledge to kind of parse it and be your compiler.
SPEAKER_02:  That's actually a good analogy, because when you put it like that, if you're, let's say, you know, in use mathematics or, you know, not a better or a seasoned better, then you can go look at these, you know, very, very elegant proofs. And it's just like reading. You know, if you're reading someone's book that's very edited and very seasoned, that you're going to pick up on their techniques and everything about like their style and what makes, you know, what's saying is a really good book and you're reading that book and you understand it. You can do the same with math and you're reading these proofs from very advanced experts. Let's say, like, reading the same, or my background theorem, like, how do they get to this conclusion? You can read the whole process and how that mind works in that sense to get to that discovery. And if you can understand that for everyone, then you will eventually pick up how they think. And then you can kind of form with your own style, your personality.
SPEAKER_01:  Totally. So we see that happening in code too all the time, right? Yeah, yeah. So there's a joke, like, the best way to understand, like, what goes on in somebody's head is to read their code.
SPEAKER_02:  Like, there's not going to be documentation all the time. So you got to read the code. That's actually the documentation because it represents how they think and how they're going to communicate their ideas. So now I didn't think of this before, but that just makes proof right, even more important to me. Reading very advanced proofs will get you to very advanced thinking. That's actually, yeah, I just didn't think of that before. So now I'm even more set on reading proofs. Yeah. It's just like reading a book and then picking up their ideas, translating that to math, reading a poem, reading up on their ideas. Yeah. Well, that's a good way to understand the episode.
SPEAKER_01:  One other kind of interesting thing about the proof writing is at first look, like, it's pretty common for sometimes proofs to just sometimes like they don't have a whole lot of symbols in them and you're like, oh, this looks approachable and they start reading it and then your mind gets like just bent and like just demolished by all the sort of logic that's happening in the proof. It can be a little, yeah, if you don't really like train yourself up on proof writing, you just try to like open up some math textbook and try to read some proofs that don't have too many symbols in it. It can still be like pretty rollercoaster experience. Again, back to proof as a coding language. It's a very esoteric language and it's often very esoteric in subtle ways. There's various forms of language that have very specific mathematical meaning but it's also used in conversation to like more and more loosely. And if you are reading a proof with conversational language in mind, then you might start getting confused. Anyway, more justification for going and learning the proofs instead of just hoping that you can just like jump into the deep end and swim.
SPEAKER_02:  I wouldn't even say, even if you're not reading proofs, all these math books, they have a speak about explaining the math in their sentences and you don't want to like sit there and be like, oh, what does a symbol mean? You just kind of forget the sentence you just read and you kind of stuck on this and then you have to read it again. So it's like wasting time as well, doing things twice. So I think math is a language and just like Japanese or Mandarin, the symbols represent. They also have like the speaking words. So it's like kind of two languages and that's kind of what math is like. Yeah, I guess it's very similar to Japanese.
SPEAKER_01:  Totally agree with the whole like, yeah, if you're trying to read math and you're getting continually interrupted by like, oh, what does this symbol mean or what is this phrasing mean? Like, there's no way you're going to be able to see like the forest for the trees and the whole proof. Just like reading. If somebody doesn't know the definitions of various words or somebody has a level sounding out where it's like, there's no way you can read Shakespeare play if you're having trouble with the individual words.
SPEAKER_02:  Yeah, exactly. And so, you know, you have the set of areas like a large component of this and if you don't know, like symbols for an intersectional union, you know, subset, element of, etc. And when you're reading through like this kind of logic and reasoning, when they embed, you know, these symbols within words, then you're not going to read it fluently and then you want to understand the whole thing. It's kind of like, yeah, reading a book, right? You're going to forget about the kind of story and be fixated on, you know, what does this go down the tangent of like, oh, go and chat GBT and type up, what is the symbol? And you just spend like five minutes trying to understand this one fucking symbol and you have to read the whole fucking thing again. So it's, I think it's super important. And I think, actually, I've heard that a lot of math students don't actually study proof writing for a long time. And that holds them all back into like later on the road.
SPEAKER_01:  Yeah, that was something that I've seen just over and over again is those people who they come out of high school, they are solid on their mechanics of calculus and they think they're all set for college. And they're really excited that they maybe want to do a math major and they get into some really serious university and they sign out for the hardest math courses there. And I mean, and they were like at the top of their class in school, did really well on the AP, or IB calculus or whatever college credit calculus exam that they take. And then they show up to university and they just get their ass handed to them. All these, what happens is like, there's this level of mathematical maturity that is often assumed for students who are coming into a math major, especially the ones who are like really excited about it or like pushing as hard as they can to take the advanced classes. So get yourself into a class that has some proof writing ability kind of baked into the prerequisites. And you've never actually gone through, you don't know what a proof is, you don't know the union intersection, you're just not comfortably fluent with all these proof mechanics. And you can, I'll just feel like you're drowning. I actually talked to last summer, I met somebody who was telling me how they were basically in this situation at the University of Chicago and they entered as aspiring to major in math. And, but then what happened is, yeah, they just, they got their ass handed to them because they didn't have this proof writing foundations. And it just, it felt like there was just a worth of knowledge that a lot of their classmates were coming in with these classmates being other people who are also very intensely interested in math and who have often done a lot of reading outside of the normal school curriculum. So they've not only learned their calculus, but they've also like, they know how groups go. Yeah, they know they probably take in some linear algebra, some multivariable calc. That's very advanced. And yeah, so the courses are kind of oftentimes these university courses are kind of built to cater to that kind of group of students and they're not going to slow down for students who need to learn what a unit or intersection simple is. But anyway, so this guy ended up actually switching majors to physics. Not because he enjoyed physics more, but just because it felt like the playing field was more level there, where like the beginning physics courses, there's a little less background knowledge that that seemed to be required into it. Like a lot of the physics proofs derivations, they're kind of more mechanical. And like you start with some formula and then you rearrange symbols to get to the output result. Because like in pure math, a lot of these proofs are not just symbolic manipulations, but they're also kind of the involved logical rules of inference that can get a little. But yeah, anyway, this guy's overall reaction was just like, man, I wish that somebody let me know in high school that there was this game being played and proof knowledge is part of that game. And just because you do well in your high school curriculum, like that does not mean that you are set up to be at the top of your class in college. The people who you are competing against or collaborating with or whatever, however you want to think about it, the people who are sitting in class next to you, yeah, they're going to have a lot of this other background knowledge that comes mainly from proof writing. Yeah, so it's totally one of those things that's super good to learn before you go to university. I would say that just to get back to what you were saying originally, like I think people who could graduate from a serious math major, yeah, they will have proof writing experience or at least they really should have proof writing experience if that's a serious major. But a lot of people get filtered out of the major, get weeded out just on the basis that they didn't know proof writing originally and they found it hard to kind of spin up on their
SPEAKER_02:  own. Yeah, it seems like it's, you're trying to program without understanding the language is what it seems like or like trying to write a fucking novel without knowing English. Yeah, that's what it seems like proof writing. Yeah, because it's just the punctuation, it's the wording, combining words to create sentences. It's like the foundations of language is what it seems to be. And for some reason nobody wants to learn the language. It's just like, I guess they see it as unapplicable, but it's really applicable to every part of math. Like, yeah, I don't know, you can't avoid it, I think. It's kind of weird how people do it.
SPEAKER_01:  Right, there's a set manipulations that you were talking about. Like, yeah, they show up every pure math, applied math. It just, it doesn't even matter. Like, other fields that you're right. Yeah, yeah, exactly. It's just part of the, it's part of the, yeah, I use it right.
SPEAKER_02:  Yeah. Like any kind of quade was just a set manipulation, really. Yeah. So that's what it seems like. Hmm, it's really interesting. You know, your kind of journey was interesting though. You went from computer scientists into like a math software dev, like math academy. So what was, what was that kind of transition like?
SPEAKER_01:  So I actually came that before the computer science, I actually came from a math background. So, yeah, so I, my journey, I was in high school, I was just, I was super into math. So I was, you know how I described like, there's some people, some skins in high school just like super crazy about math, learning linear algebra, multiviral lookout, proof writing, everything, like not stopping at calculus, but that was you. But yeah, that was, that was me. So, yeah, so I self studied pretty much, I'd say like three quarters of an undergrad math major during high school through MIT OpenCourseWare. And I got to college, I majored in math and that's, at the same time, that's when I started getting into coding more seriously. I started working in data science and then a couple, well, about a year after graduating college, that's, that's when I found math academy. Yeah, that's, that's where I got involved. And initially in math academy, even, I actually got started way, way back in 2018, we were making some videos actually. We don't do video now, but we were experimenting with that at the beginning. It was one of our kind of the first form, the first conception of kind of like a guided lesson for, for math academy involved a video. And yeah, so I made like hundreds of these kinds of educational videos for math academy and I was writing a lot of content. It wasn't until 2019 that I actually started working with Jason on the code side building the automated system. So, so yeah, I actually, I had it a lot easier compared to a lot of people who started out in computer science and want to make the transition over. For me, it was more, it was more of like, I, I, I came from a math background and had to make the transition over to computer science.
SPEAKER_02:  I see. I vote. Yeah. So, I'm not sure if I'm going to go to the computer from like MIT because a big question I have is like, okay, you, we're programming. It's very easy to, to know where you're going next after you finish a project and like load a topic. All right, you go expand to another thing, you build a project that wants some other stuff or add another file that connects to the main file. And that's how you learn some other things like, like the multi-threading from like single sequence to multi sequences in parallel. And you learn about one shot and all this other stuff. But with math, for me, it's unclear because I, what I was about, I've never learned a language before. And, you know, so many branches, et cetera. But how do you decide, well, how do you even go beyond your skill level? You like, read more advanced books. Do you, yeah, it's just kind of unclear. Like, let's say someone finishes Math Academy, for example. What do they do next? Like, do they just go try read some advanced book that they may not understand and then try and figure out how to learn from books and math? Or, yeah, what's the next steps?
SPEAKER_01:  Yes, it's a great question. Yeah, well, I think, okay, so, yeah, initially, highly structured curriculum is just vital. And that's, yeah, that's basically Math Academy takes from fourth grade all the way up to university level math. But yeah, what do you do when you get out of the other end? And so I think it all comes down to like, what, what are you trying to accomplish? So think about math. I mean, as, as you know, as we're talking about not too long ago, math is just very, very branching subject. So as you're, as you're working your way up through the foundations of math, that trunk, the tree trunk of knowledge that is just so core to have like with the set manipulation of proof writing, calculus, algebra, all that stuff. Once you kind of hit the more, the higher point of like having pretty much a solid undergrad math education, you kind of get faced with an existential crisis. Like, what are you doing next? There's not, it's like the, the, the trunk has turned into branches that are branching out so much that you, even if you are going to be the world's greatest mathematician, and you are, you just have no chance of being the best on every single branch. There's too many branches. You have to start making a choice of what you're actually trying to do. And that's, I think that's when it becomes a little hard for, for people who have just followed the standard curriculum without really thinking a whole lot about whether interested in beyond it or why are they learning math? Because you get, you get faced with this fork in the road and it's actually like 20 different forks, 20 different paths that you could be choosing from. And you have to make a decision. And so if, I, so I, generally what I, I recommend is, is like, yeah, you got to be building up your foundations, but at the same time, it's always good to have kind of in the back of your mind of like, okay, what, what am I, what is my goal? Like we talked about them in the past episode. What is the goal? What is my goal of learning math? What am I trying to do? Is there a particular type of field of computer science where I'm trying to, trying to write some autonomous agent that does something or do I want to be an academic mathematician who moves theorems on prime numbers or whatever? It's like, it's kind of got to write, just decide what, what do you, what do you want your future to look like? And then, and then that's when you start kind of finding the right thinking up books in that direction and working on projects in that direction.
SPEAKER_02:  Right. So I guess when you finish the trunk of math academy, every field relies on this. So that's why it's like the trunk. And then once you finish it, you should be able to pick up any book and just understand it. Is that kind of what happens?
SPEAKER_01:  I'd say loosely. Yeah. Now, so it's kind of like, see, loosely, I'd say that's correct, but there's also even, even when you have this trunk of knowledge, there are, there are, there are books and resources that are, that may be too specialized. It's still a really old school. Right. So it's kind of like you have to scaffold yourself up into some subfield, sometimes do some, some introductory, some introductory material.
SPEAKER_02:  We understand the introduction because you have the foundations, right?
SPEAKER_01:  Yeah. Yeah. Right. So like having the mathematical foundations puts you in the realm where you can actually read some introductory material into some niche field of math.
SPEAKER_02:  Right. And then you get to the point where you can reset again.
SPEAKER_01:  Yeah, exactly. Right. The thing that makes it a little tricky. So in the lower levels of math, like, okay, elementary school up through calculus and, and even university level math and math academy is like, this is just scaffolded super, super well. Like you don't, you don't have to worry about what resource you're, you're picking up or what order and what you're working problems or reviewing thing. Like this is all taken care of for you. So you just, you just get on your computer, do your XP, solve your problems that we tell you to solve and you're all good. But when you get, when you're done with, if you learn all the math of the system and how it's time to like get more specialized to some self field, you end up have to start managing your own learning process. And that, that is, that's, that's kind of orthogonal to the math knowledge itself. It's, it's just knowing how to keep yourself accountable for learning, making sure you're actually solving problems, doing some projects, scaffolding them up in difficulty, not, not starting with something that's too hard, starting with something that's just the right level of easiness. Yeah, it's a little, a little challenging. And especially as you reach the edge of a subfield, the resources that you have at your disposal to teach yourself, they just get worse and worse and worse because as you get to the kind of cutting edge, that's like, people are just doing, they're just publishing research papers. That's it. Like, they're, that's what you have to learn from. It takes a lot of time for, right, the cutting edge to, to become known and for known things to become well scaffolded in a current.
SPEAKER_02:  Right. Yeah. So like the, the further up you get in the branch, you get spinner in the finner and then the resources are just shitter and shitter. So it's harder to learn.
SPEAKER_01:  Yeah, exactly. Sitter, thinner shitter and shitter.
SPEAKER_02:  That's all right. Then you just kind of, but this is such an important skill though, because like you have to direct your own learning to get to a cutting edge. And I think it's like, well, maybe it's like programming, but I think we're programming you can just kind of keep building and then optimizing and then you're kind of at the edge. Whereas like map, you have to learn from others that are really better communicating. And then you have to kind of build the next level from your own, you have to be a creative thinker at that point, because like no one's there to help you. You have to do yourself one.
SPEAKER_01:  Yeah. Right. You kind of like, yeah, in any field, once you reach the edge, it becomes about creative production, which is related, but also very different from acquiring no skills when you're creating the new things.
SPEAKER_02:  Yeah. Totally different game at that point. That's super interesting. Yeah.
SPEAKER_01:  You know, what kind of interesting. The first time this happened to me actually, when I learned a bunch of math and I got really into a subset of subfield machine learning, a particular neural nets. And at the time, this is back in like 2014. So convolutional neural nets were like pretty much the cutting edge machine learning that then I don't like there's tons of courses now. And like this is how you do convolutional nets. And like it's very well, if you want to learn convolutional neural nets, then it's very easy to do so nowadays. But just just 10 years ago, the resources available for learning that kind of stuff are just so sparse. And yeah, so I kind of, I got really interested in machine learning neural nets and I kind of like kind of hit the edge or like it's, it felt like, like trudging through tar. It's just so hard. It was just, yeah. After learning like, you know, MIT, open course are great calculus course. Yeah, videos like, okay, like calculus differential equations, probably statistics, like there's tons of information about this online. Now, I mean, obviously it's not still not amazingly scaffolded compared to other things like math academy, but it's still like you can still, if you're motivated, you can make good progress. Yeah, once, once I kind of hit the edge where there was like, it was just getting really hard to even find like real like textbooks on this. Yeah, right. It was like walking through tar and I actually had sort of that existential crisis and I'm like, is this really what I, what I'm interested in? It turned out the answer was actually, no, I kind of got tired of trudging through that tar. I was just not committed enough to, yeah, it seems like neural nets stuff.
SPEAKER_02:  When you go through a niche, you really have to be interested to just like explore and like be creative of wise, you just kind of burn out because this is no help. So yeah, it's not a way to think it's like a truly, are you passionate about it to like continue doing this?
SPEAKER_01:  Yeah, it's a lot of interest becomes, yeah, very important, right? Because that's kind of what motivates you to keep going despite the tar. Yeah, you have to say like, okay, yeah, I'm charging through tar, but I just, I love this thing that I am. I love the tar. I love the tar. That's basically, yeah, if it has to be worth it. Right. It's not just about moving fast and acquiring knowledge at that point. It's about really caring about the subject.
SPEAKER_02:  Yeah. I think this is why I started like preaching that it's going to actually start to enjoy mathematics and really love it and see the beauty and things because I feel like learning first thought, I've never had a language before. If you don't classify like programming languages, I just think it's just modified English. But like, the only way to really motivate yourself, at least for me, is what I see is like, you just got to really try and find the beauty in these fucking shapes and random air symbols. And then I don't know, I don't know how to say it, but like, I don't know how people can just be like, quote unquote motivated all the time to just like pursue something about trying to fall in love with it. Because like, why continue them? If you're not going to be, I mean, people can have goals, right? And be like, oh, you know, help me do xyzad, but for a grand goal, I think this is where like the, that like quote of like, fall in love with a journey and not the destination comes from. It's like, if you really enjoy the journey, then there is no destination really. You're already there. It's just, it just continues. And then you find yourself somewhere like the edge of chaos. And then you're now pioneer in some sense. So that's how I kind of started to view math is like, okay, we'll just do it every day. And I've seen this happen as well. You know, if you spend all your time with like drug addicts, you're going to become a drug addict, spend your time with millionaires all the time, become a millionaire. You just spend time with math all the time, you'll probably be a mathematician. It was something in it, right? It's just like, can you open yourself in it? And the hardest thing to do is really just start each day. Because once you're already started, you get in Swaflow state naturally and then you Oh, six hours gone by, 50 days, these, but like this is the greatest. Yeah, it's just super, even with like anything, but it's just starting as the hardest day. Like when people make new res, new years resolutions, it's like go to the gym, sign up two weeks later, you're going, or like, you know, you help me any of the, like any basic thing. There's like some weird phenomenon when people just like had insane motivation to the star and that it just like tanks to like, glows your road. So from like your experience of teaching and even like learning yourself, how do you, how do you like, stop? I know it's super simple, but is there any like structural approaches to easing, opening the gate slightly bit more to have it easier to start? And yeah, just like make it as easy as possible. Start aside from just like, I'm going to start, you know?
SPEAKER_01:  Yeah, yeah, I would first, I would totally agree that it's like, yeah, oftentimes in order to really fall in love with the subject to get that passion. It's often not love it for a site. And you really have to, right, you have to spend time with it. And you have to develop a habit of spending time with it. And so, right. So how do you really get started with something if you may not have that passion at the beginning? Mind you, it's also very hard. Yeah. So, yeah, and it's okay. So right. So not only are you not in love with that at the beginning, but it's also very hard. It's asking a lot of you. It's very taxing. How do you get started? I think, you know, I-
SPEAKER_02:  You know, my words is unknown. Well, just to make it the hardest possible story. Yeah, okay.
SPEAKER_01:  Yeah, no, this is right. Let's state the realities that play here. Yeah. Right. Reward unknown. How hard is it? Very hard. How much do you love it? Not at all. Okay. Yes. How do you, given these conditions, which are ultimately very unfavorable for you, getting up and doing this thing, how do you get yourself to do the thing? And I think the answer to that is just to do, just, I guess, to use math terminology here, you do epsilon amount of it. Just like reduce, find ways to reduce these barriers. Like, okay, it's very hard. Anyways, to make it easier. Okay. Just make it easier for yourself. You don't have to do it for a long time. Just make it, I guess if we're talking about fitness, like, if you've not been exercising for months, like, just sitting on your butt all day, not doing anything, how do you get yourself on this track of getting fit? I think day one is not actually going to a gym and lifting weights. I don't think it's going out and running five miles. I think day one is just going for a walk, going for like a 20 minute walk, 10 minute walk, five minute. I don't, whatever, five minute walk is like too difficult for you. And you're like, oh, I don't want to do a five minute walk and do like one minute walk, do like 10 jumping jump. Like, however low the bar needs to be made for you to start in this direction, well, we're as far as far as you need to. And then just make sure that like each day the bar is getting a little bit higher. So let's just say, like, say we've got somebody who is so lazy that they like a one minute walk is a one minute walk sounds like that's enough to dissuade them from getting fit. And then just make it a walk to the end of the room and back. That's 10 seconds or whatever. Then like the next day do that twice. Then the next day do that four times. Okay. Now you're probably like at the point where you can be walking for minutes. And you kind of just like just do it a little bit, even if it's like such a small amount that it doesn't feel like you're actually doing anything. And you just you compound that you multiply that little bit a little bit higher every day. And then you kind of just get yourself into the groove of it. And before long after the first week of this kind of routine, like just get up, walk across the room, come back, sit down, maybe instead of just doing that once a day, you do that a couple times a day intentionally. And then you get to a point or a couple days later, you're like, okay, I'm walking around my room. This is kind of boring. Why don't I just at this point I might as well just sit down and walk around. And then at this point I might as well just like go outside a little bit and go for walk around the block. Okay. Now you do them like a five minute walk around the block. Maybe the next day you do that again, but you go another block down. Now you got a ten minute walk. And then you're like, well, I'm just like walking for ten minutes here. Like this is getting a little boring now. It's like, well, okay, why don't you run that route instead? Okay. Now you do a run. It's like just a two minute run of that. And then you scale it up, scale it up, scale it up each day until like soon enough you're doing like a ten minute, twenty minute run. You're covering like suffer miles. Okay. Maybe you get tired of running after all. You're like, well, okay, I can allow myself not to run if I go and like do push ups and then some pull up hangs if you can't do like full pull ups, but like some kind of calisthenics. Then you do that. I think that's the trick for anything. Not just fitness, but for even for math, for getting into like, so somebody is looking to get into math specifically. I'd say like, okay, first step, if you don't want to start with problems that are at the edge of your ability, just solve a couple math problems on something that you know. If it's like arithmetic, like just say, even if you don't know like fractions or decimals or whatever, just I don't know like come up with a or just look at it. Look at even if it feels a little silly, like just look at some multiplication facts or yeah, your times tables or some fourth grade material online. Just solve a couple of those problems and just get yourself into the mode of like, okay, this thing feels easy. I feel like I can do more. Okay, I'll do more next time. And then even if it feels a little silly at the beginning, like if you just keep compounding that, everyone knows how compounding works, you get this exponential growth and the amount of work you're doing. And you kind of, yeah, you trick yourself just to start out with the easy thing and then just develop comfort with that easy thing. And then just, yeah, just explode the volume of work, honestly, until you're doing serious workouts.
SPEAKER_02:  By the analogy, we look at gym, it's like when you go into the gym and you're first starting, it's super shit, but then you feel like the muscle where it doesn't hurt anymore and it gets kind of easy and you don't want to lift the really light weights. You want to be on like the edge of chaos, which is just, you know, just below your max, but you can wrap it out. So it feels, you still feel the pump and like you're still trying and then it's like, okay, cool, we did that. And then the next day, you're actually a bit better as your body adapts to it. And so you go a little bit heavier and it's just like infinite blue pull of that, as long as you keep the habit. The other thing really is just like keeping that momentum, which is the habit and like the routine. Like when I was, when I changed my schedule to waking about 5 AM and like skipping instantly in the morning and then going shower and then I did, instead of like 30 seconds of like a cold shower, do 30 breaths, like a deep breaths and now it becomes like a routine. And it's like creating the routine where it's just always repeatable and you have to think is like the main thing. But I think like even with math, just start off by potentially just like doing multiplication for one minute. Like those are, you just basically got to make it as easy as possible to like get the motivation to continue. Like when I start skipping, I skip for one minute and like the hardest thing is literally just picking up the thing. Because when you already have it in your hand, you want to swing it around and you're like, oh, well this is kind of boring. Let me jump. Yeah. And then, you know, like, oh, put some music on the back or it it. So it's really just like, how do you get in front of it? And like this sometimes when I'm just trying to like recently, I've been trying to just do math, I still the computer. I look at like there's a very, it's like at the edge of my knowledge, which is like differentiation right now with like, I don't know, multiple things. Like anyway, it doesn't matter. So it's like something at the edge of my knowledge. And I just said that I'm like, my brain does not want to do this right now. And so I just didn't do it. But I think if you just, if I just did like, I know a minute of multiplication or maybe like five minutes of just doing the same shit all the time, you're like, okay, well, my brain is kind of warmed up and I need some more stimulation. And then you, you would chase the stimulation of something harder. And like shit, you know, like addictions, like, I don't know, things that give you like really big dopamine boosts. I don't know, like junk food or like porn or anything similar to that or like social media. And I think they just destroy your wiring in your brain where you have such a high requirement to get dopamine, which makes it harder to do the actual important things because they don't give you as much dopamine as these like hacking things do. So I think a big, like thing to actually do as well is just cut back on the bad things because usually people that are, that are in places you want to be don't have more than you, they actually have less than you, you know, they don't, they, so basically what I'm saying is, you know, they don't watch porn, they don't go on social media as much. They don't do all these bad things. So actually they just have less than you. So then the solution is just doing less of the bad things. And it helps because I'm at like, I actually tried this and I was just sitting in a room, like blackout room, just sitting in a chair staring at a wall to like test what like boredom would do. And like you put a book next to you and you just get so bored that you would just do the hard things anyway, like you would pursue pain because you would just have no stimulation. And so what I've been doing is I've been going to bed two, three hours without any blue light, so therefore no computers. And I just put a book next to my bed, which is being the proving book. And I'm just like, well, I'm bored. Let's just read this fucking book. And I think it's important to not have over like too many options. If you have like 10 books, you're just making it harder to do the thing. Whereas if you just had one book, you would pick up that only thing. The only one option, right? So get so bored and give yourself one option. And then you're forced to do the hot thing is the internet. So if you're trying to force yourself to do like Mac got away, it's easy to just type x.com, which was kind of a genius move of Elon instead of Twitter. But like, that's the hardest thing is like this restriction on the internet. I mean, you can even use like some web blockers, website blockers, but then if you're going to be an addict, you're going to be a good addict. So you find ways around it, you know? Yeah. So I don't know. It's kind of difficult, but what I've noticed, it's actually just removal of bad habits, which makes the good habits easier. And then they kind of replace it.
SPEAKER_01:  Yeah, that's a really good point. Yeah, because I guess right. It all it kind of does come. It's not just about how easy something is in an absolute sense. Kind of like, what is it competing against? What other actions that compete against? Right. So it's like, you can try to make whatever thing that you are trying to make yourself do. You can try to make that that easier, but ultimately you are putting yourself in an environment where it is very easy, even easier to do another thing that you know you're not supposed to do. Yeah. Then you're just shooting yourself in the foot. Right. So part of it is just setting up your environment where your choices are kind of more minimized or like where you're kind of manipulating yourself into taking a certain action, I guess. It's like your past self is manipulating your present self.
SPEAKER_02:  Yeah. So for example, if you're out, I don't know, like a very specialized business or everyone's in suits, you know, dinner, you're putting yourself in this environment where you're not going to just, you know, like scream out loud because it's going to have some ramifications instantly. Yeah. You're in this environment where you kind of like social norm is to be a certain way. Well, you can apply that to anything else. Well, you have to create the social norm to be a certain way, which would be, let's say, math, if you're in a math class, everyone's talking to a math, therefore you got to do math to compete and stay there and like just feel a part of the norm and no standout kind of. That's how I kind of see it. Obviously that isn't quite the art to do. So it's just how do you, yeah, the question really becomes how do you just remove bad things because then good things are left. Right.
SPEAKER_01:  So yeah, I think that's, yeah, that reminds me of like how these kind of like social like study clubs often, often work well for people where it's kind of like you go into some like online, some discord where they've got like some study session going on. Everyone is like doing math for 20 minutes and we're going to chat for five minutes and math again for 20 minutes. It's like you, there's the other's kind of like a norm in place and it kind of like funnels you into this kind of behavior, even if it's not something that you would have wanted to do by yourself.
SPEAKER_02:  Ultimately, it's just how do you, like if you just compare yourself to someone that you want to be, that person doesn't even have to exist. It could be like future you or something. Yeah. What does that person not do? And then you compare yourself to that. And then you, and then like let's say theoretically you got rid of all those things that you're not meant to do and like how much better would you be? And that's actually I think a way better framework than just figuring out what you'd have to do. Everyone knows what the fuck they have to do. Yes. Everyone knows deep down inside, you know, you're meant to do the good thing right now instead of the bad thing. So it just comes down to like how do you not do the things you're not meant to do. Which is very like I think counterintuitive really for the normal way of thinking in like at least in this generation. Anyway. So I actually read your blog as well. You mentioned you're mentoring like a math prodigy and I'm curious of what makes them stand out from like the rest, like what makes them a prodigy. And is this something you can like learn a gain over time unlocking your inner prodigy?
SPEAKER_01:  Yeah, that's a good question. So unfortunately the answer comes down to this particular kid that I have been working with for a long time. He has these these just individual differences, these that are advantageous for the domain of he loves math. He's loved math from an early age. Super interested in it. Actually the way that I came in contact with him was back when I was doing more tutoring five years ago, six years ago actually his parents, I came in contact with his parents as parents were looking for a tutor to just keep him occupied for a couple hours each week because he would be doing all these math experiments, like arithmetic experiments. And he'd be basically just doing all these like little mini projects on like properties of numbers. And so he would just be talking to this, his parents about this all the time, non-stop his parents. He was like, how do we like somebody has to like take him out to play mathematically? Like you know, if you guess the kid is so amped up on math, he's like, she's just called them. And it's like, I guess if you think about like how kids are around like dessert or like marshmallows or whatever, like you just give them too much sugar and I'm like, okay, like this kid was a little like that with math. So I would just kind of like, yeah, work with him on like, oh, did you like, that's a cool experiment that you were doing? Did you know that like, this actually relates to this property in arithmetic and blah, blah, blah, blah, here are some problems that we can do to explore this further, naturally led into like algebra and calculus after that. And we have lots of various wide ranging fields of math, but the idea is like, so he did the benefit of just being very intrinsically interested in this from a young age. I don't actually know. And I don't think he knows either why is he so interested in math. Why did he fall in love with it? So we're like, so that's the remaining question. Why is why is that? I'm not sure how or if like that can be replicated in somebody who does not have the love for the subject. Well, I mean, there are people who who fall in love with math later on, but I'm not sure the same process that that led this kid to get so immersed in math. I'm not sure if that just comes down to like just the way his brain was initialized, basically or if there was some process that kind of like led it into this love for math. I mean, another advantage that he has is just cognitive differences. I know a lot of people get uncomfortable talking about things like IQ and whatever, but people do have differences and say like working memory capacity, generalization ability and forgetting rates. And these can also be like domain dependent too.
SPEAKER_02:  So it's like people like a naturally better at some things than others.
SPEAKER_01:  Yeah, yeah, yeah, exactly. So yeah, now everyone's brain is the same. And yeah, people have advantageous just like people have physical advantages that lead them to certain sports. Different phenotypes. There are. Yeah, right. Same thing cognitively. And so this kid just has like very high generalization ability, particularly in mathematical and quantitative reasoning. Now his forgetting rate is shockingly low. So the way I would describe working with this kid versus versus a lot of other like very bright students that I've worked with, but they're not like at the same kind of prodigy level is this kid like you can go through some problems with him, introduce some new material to him. And it's kind of like, it's almost like reminding somebody who has just gotten rusty on some material. It's weirdest feeling, but it's like, you know, if you get like rusty and your math from high school or whatever, then you like to have a textbook and you're just kind of like, oh, yeah, I remember how this was back. And now like, now imagine like, if there's like a, now imagine that you had never seen that before, but you just had like from your previous, whatever you've learned in the past about math, you have internalized it and generalized it well enough that your, your knowledge base is refined that integrating this knowledge becomes a process like refreshing old memory almost, or just those kind of fits right into place. So that's kind of, so that's the speed of knowledge acquisition. And so typically, I don't know how much you've done like tutoring or teaching before, but if you work with a typical student and cover some things in a session and then you wait like a week or two weeks and then you want to continue building on that knowledge, typically for an average math student, all that knowledge is, is has almost a K to zero. You basically have to reload something. It goes, it goes faster the second time, right? But you have to go through the whole process again. And if you have a really, really bright student, then they might remember quite a bit and you don't have to spend as much time re re teaching. But with, with this student, it's like the memory still fades a little bit. It's not like it is just locked into place for eternity. But it is the rate of decay is just so slow, so slow to forget that that is just, it's essentially negligible. You can wait a week or two and just kind of pick up where you left off. And that's not very much friction at all. Right. So there's all these factors are just compounding into an accelerated progress through, through math.
SPEAKER_02:  And so, like really lucky dive stroller genetics, basically.
SPEAKER_01:  Yeah, it's like a lot of, really lucky role plus a lot of leaning into his interests and a lot of hard work. Yeah. None of this would happen without the hard work portion. But it's like, it's a lucky role where the hard work is actually kind of like, yeah, I'm sure a lot of that is also just because of the like building up the habit and, you know, so good at it being so in love with it, that it's part of his identity. This is who he is. Yeah, yeah. That's right. There's all these helpful, full factors that can definitely be replicated. But that kind of, I guess, so I guess the two things are like the two things that are hard to replicate, just like take any person off the street and give them to the same level, is like, how do you, how do you instill a love of mathematics that early? And how do you, like, they probably don't have as advantageous cognitive differences for mathematics? So I think, I think that the hard answer is like, most, pretty much all math learners are experiencing a lot of friction during the learning process. And there are ways to greatly reduce that friction by trying to develop more interest in the material by doing review, by doing mastery based learning, space repetition. All these things are kind of reduced friction in the learning process, developing a habit. And so people can accelerate their math abilities well, well, well beyond like, what is the status quo? But to reach, to get to like that kind of prodigy level, I don't know, I think it comes down to kind of like you have to find an area where you just really like where you, yeah, you have competitive advantage in it somehow. Interesting. Yeah.
SPEAKER_02:  So you could also develop that interest very deeply if you find something you really like about it. I think maybe it just comes down to like problem solving because I jump from, like, I mean, I change my career multiple times. I was here at crypto, I bring it to trading, to know, math and AI and robotics and biology. And I've done everything, but I think it's all encompassed in the one thing which is just problem solving and wanting to know something really deeply that's uncovered. Transcribing. Transcribing very well. Yeah. So maybe there is some like generalization and then also thinking creatively of like, I know linking shit together of like, you just want to hit stuff together and make it compliant. And that's at least my traits. And like, I don't know, I think being very persistent takes you a long way anyway. So resilience is something also that's learned. But you can go far with just like very basic stuff that you can just learn. Like, I mean, creativity is a skill. You just combine as much knowledge as you have into one thing basically from different fields and it's not sectioned off. It's like an open gate. I can just interact with anything. I think that's just a skill. You just use a book and write your shit down and link it all together later on. Bank don't. Creative don't.
None:  But yeah.
SPEAKER_01:  You know, I should say I think engaging in these like even if one, even if a person may not necessarily reach the level of prodigiousness of a particular person that we're comparing to a great in the field. If you pay attention to what things are fueling somebody who's great in the field, like just consistent practice habits for efficient practice habits does not mess around during practice. It's intense love for it. If you replicate those things in your life, then yeah, you can definitely go very, very, very far further than like anyone would ever expect. Oh, yeah. Yeah. My question, like if you're just like have, if you don't have any other like competitive advantages in terms of like cognitive differences, whatever, there's, I mean, there's still a question of like, okay, how good will you get? Probably not as good as the best in the field, but you'll still get pretty damn good.
SPEAKER_02:  It's not even playing field. It's like, I mean, if everyone did the same amount of work and have the exact same lives, then genetics would play a giant role. But like, even some people without a genetically superior in some, some ways just don't have like these basic fundamentals down of like resilience and persistence so that the magic is pulled behind people that just have those things that that may not be as genetically inclined, you know. It's just, it's not really a giant factor. And then these like, I guess the prodigy is like the hard work plus the advantages combined. That's what makes them so great. But like, anyone could really just follow hard work super hard and get to a place where maybe even people with the lucky dice roll of the decks don't get because they just totally
SPEAKER_01:  move. Right. Yeah. It's the saying like, what is it like hard work beats talent when talent doesn't work hard.
SPEAKER_02:  Yeah. Yeah. But then you have those, I mean, any other people that are like, insanely, you know, like six or seven and you can just like do everything. Oh yeah, that's kind of like, it's interesting though. It seems like it's just a really deep passion for and want to learn, which I think gets the compounds that better you get as well. Like the more, the better you get, the more you want to understand and the more you don't understand because you know that there's so much and that makes you want to go even more. It's basically just finding positive feedback loops the other day. Yeah. Yeah. Anyway, that's right. That's exactly right. Let's, let's, let's jump over to like the AI side of things as well as meant to do for this episode. So with math academy, you are the mind behind the AI system, a quote, quote, quote, AI system that presents the reviews, the lessons, the multi-stirp, space lead, the, the lessons to do at a specific time instead of giving you a curriculum like you traverse that manually. It presents it to you with these knowledge graphs. And if you don't finish, you know, if you fail lesson, or you don't really get a review, it pushes you back until like a prerequisite. If you fail that, go back, pass it, go up, etc. So a few questions, but how did you, how did, well, I think, but yeah, just explain it briefly first and we can answer other questions.
SPEAKER_01:  Yeah, yeah. Sure. Right. So this is kind of just the ultimate goal of the system is to get you learning material in the most efficient order possible. And there are things that are, that are obvious, like do prerequisites before doing the harder stuff, but there are also things that are, that are less obvious, like how often should you review things? Well, it kind of, it kind of depends how, how well you did on them to begin with, how strong you are and also how much, how much review you're getting from other things that you're doing. So if you, if you are, say you're solving, say you're solving a, a linear equation, say you're solving like a two x plus three equals four kind of equation. Well, you're, you're getting implicit review on, on something like two x equals four. You're getting like, like eight, a two step equation encompasses a one step equation because that's a skill that you're doing underneath. So, so you, you got to take that into consideration when just deciding like how much, what do you need to review? If you do a two step equation lesson, then well, that counts as your review of your one step equations because you, you did that as a sub skill. And so the, and it also comes down to like, okay, well, there's all these nuances with review that you have to like kind of track through the knowledge graph or the prerequisite encompassing of mathematical topics. And you also need to be picking tasks that are leveraging all that information to short an efficient course, an inefficient path through the material. So not only, not only tracking that information, but also making the active decision like, hey, this student has these five topics that they need to start reviewing pretty soon because they are in their space repetition cycle. These reviews are coming up. Hey, there's this lesson that they know the prerequisites of they haven't done it yet, but we need to do this lesson anyway at some point. And this lesson happens to encompass those five reviews that they need to do, or maybe it encompasses like three of them fully. And then the other two, it's sort of like halfway encompasses and it can push those reviews off a little bit more into the future to make the active decision like, hey, let's serve them this lesson that way. They don't have to spend time explicitly reviewing this material. It's all just funneled into the learning of the new thing. And so right. The overall goal of the system is it's an optimization problem. It's minimize the amount of work that the student has to do subject to the constraint that they are getting all the spaced reviews that they need and they are being put in a position where they are able to successfully acquire the new knowledge. So like doing the prerequisites before doing a more advanced topic. Yeah. So it's right. It's kind of like that's the math problem that it solves. Optimize or minimize some amount of work subject to the constraints that it's. That's it. It's an effective learning experience.
SPEAKER_02:  Yeah, I didn't notice this. Like you, I don't explicitly do old reviews. It's like, but I think it was just natural in math because like, I don't know, let's say you're dealing with fractions and you're doing some differentiation fractions are baked into the differentiation that will review the differentiation. You're reviewing the fractions anyway. And like the properties of them. So it's not like you have to go back to the basics and review them. It's like since math is so hierarchical, it's all based off the same shit. So you're just like doing, you could be reviewing like five different things. One thing. I don't know. So under like quadratic fractions, differentiation later, like it's all one thing at one point. So it's, yeah, you could optimize it quite heavily. It's a good problem to solve. But like, how did you go from like scratch to formalizing like this model of like, yeah, space repetition and hierarchical node structures? Like what was that process like of creating this and so something that you could then apply it?
SPEAKER_01:  Yeah. So this started back. Well, so for me, it started in the summer of 2019. And that was when Jason founder Ian, his wife, Sandy founded math, kind of together. And I started working with Jason in the summer of 2019. He pulled me into the coding side because he knew that I had experienced data science. And this seemed like a pretty data science problem or at least like a quantitative programming problem. Really? Yeah. Previously, I was just doing a lot of content related stuff for math academy, but he had, he kind of had this idea of math academy needs to eventually become an automated learning system. And the way that it needs to do that is by emulating the behaviors of a, of it, of an expert tutor. Now, it doesn't have to emulate every single behavior. It doesn't have to like, you know, say hi to the student, ask them how their day was, etc. But there's like their particular decisions being made during the learning process that render a tutor so much more effective than a teacher who's having to teach like a class of 30 kids with all different knowledge profiles. So these decisions that are being made on an individual basis for the student, how do we replicate that? So he, he had done a lot of reading around the subject into spaced repetition, mastery learning, blooms, two-signal problem, all that kind of stuff. So at the time, I actually was not so familiar with all these different learning techniques. I mean, I, I, I come across a lot of like, when I was doing self study, there was various principles that I figured out. Like, okay, I have to, I have to review things occasionally. I can't jump to the end of the textbook and learn the practice for this various things. I didn't have the particular like terminology for it. But yeah, he pulled me in that summer and that's when we started building this thing. And initially, initially, it was very, very simple. It was basically just a spaced, a raw spaced repetition system. There was no trickling repetition credit through the knowledge graph. There were no, we didn't have like encompassing. It was basically just, you do a lesson, either you pass it or you fail it. And if you, if you pass it, then it's added to your review cube for the spaced repetition. And you're allowed to do any, any higher level lessons that, where you, you passed all the prerequisites. We didn't have any sort of diagnostic, all the, obviously, we're starting from scratch. And we just put one student on this system. And there were a, an original student from Map Academy's in school program that had moved out of state. And so they're kind of like, kind of sad about not being able to do the, the map academy classes anymore. But we were like, Hey, we got this. We're trying to automate our system here. Do you want to try it out? And so she tried it out. And she was, I think she was an eighth grader taking calculus BC on our, on our system. And so, yeah, so she, she was our initial test student. She worked through it. Something that year. Yeah, exactly. Yeah. And yeah, it was, so it was pretty rough writing. I'll admit occasionally things would break. She's getting a lot of, she's getting a lot more review that was not really necessary. So she did a lot more work than was really needed. It's still very efficient compared to like an in person class with 30 other students. Because it was, it was doing master learning at least, it was doing that well. It was doing space repetition. It was not super efficient with the space repetition, but it was doing these things that would otherwise not be done very well at all. And so she came out that year with a five on the exam. No teacher, no, no, it was just purely working on the automated system. And that's kind of when we started realizing like, Hey, like we, we can start, we can lean into this a lot more like space repetition is like, it's a little inefficient. But there's ways that we can make it more efficient by tracking all those that flow of credit through the graph. And so we started like more seriously leaning into it, the following year. And so, yeah, following year was the, was it 2019 to 2020? Well, I guess, oh, that was the pandemic, right? So summer of 2019 was when we started working on the system and 2019 to 2020, right? And then the pandemic happened in 2020. Right. So, OK, so right at the edge of the student getting being prepared for the AP Calc test. And that's when the pandemic started happening. And so that kind of forced our hand. We were already thinking like, OK, we need to lean into this even more, add more optimizations for for efficient learning. But yeah, so the pandemic hit 2020 and all of the math academies in school classes were now going to go remote. And so we were like, oh, my goodness, this is going to be this is not going to be easy. Because like students, like we at the in school program, like we have this weakest students up from pre-algebra to AP Calc BC by eighth grade. And that is our like, that is one of our markers that they take. They very test that they take the BC test, the Calc BC test and eighth grade. And we're not going to push this further. Like that we're not going to restructure the part like we have to figure out some way to continue getting these students through all that material. Despite having fully remote instruction through the pandemic. How do you do that? The natural answer was like, well, we just need to spend the summer scaling up our system here so that we can actually run our in school classes on it. And that was a lot of work. I actually over the pandemic actually moved in with Jason and Sandy and lived with them for throughout the pandemic. We were just working. Jason and I are just like day and night every, every day, even weekends, just cranking on this, this system because we needed it. We needed to run the in school classes on it and in the fall. And so there's a lot of just we had to make the system a little bit more efficient, but we also had to make it more robust, less brittle. We were previously the one thing that we were doing that was making kind of brittle, we were storing all the space repetition records as separate database records and and that like, I mean, it worked fine until we had to make changes to the model and then all that stuff had to be recalculated and it was just it started to become just so unwieldy. So we ended up that's kind of when I previously Jason and I were working on this thing, like mostly together kind of pair coding it up and then he was just like, you know, Justin, you need to just make a a model class that just spins up and just all the students answers and just figures out their knowledge profile dynamically on the spot. We can't be storing all this crap in the in the day. It's every time. Yeah. So that's when I started kind of taking over the the modeling side. And so we we we ended up getting everything in place for fall 2020 for the in person classes that were going fully remote to use the automated system. And so what ended up happening was the automated system worked so well that that students were learning even more than before. Like during the fully remote instruction, the students doing the automated system were they were covering even more material to a greater degree of mastery than they were previously in the in person classes.
SPEAKER_02:  What was the process of actually designing the architecture of the. Yeah, it was being like a ton of I mean, like designing and I've seen like the train is a as well. Is that correct?
SPEAKER_01:  Actually, so AI that we use is it's an expert system. So it's it's based on a math company where we're domain experts on the subject of math and of teaching math. So everything that we want the model to do that we want the system to do, we just kind of pull out of our heads. OK, this is what a good tutor would do. This is what an expert would do. And this is the information that they'd be using during their decision, even something like the knowledge graph with prerequisites and encompassing records. This is all we've manually set this information. And so when you do that, you don't have to actually you don't have to train a model to figure that out. You just give it the parameters that it needs because you already have those parameters in your head. Now, this is this is really it depends on the problem being like human scale and the in the sense that like something like a a computer vision problem or something like we have to like detect all objects and images and stuff like you can't like the amount like it would take just an immense amount of like domain expert for friends to be able to do that correctly. But the problem that we're working with, the amount of information that needs to be encoded into the model, it's a lot. But it can be encoded manually putting it in this knowledge, a structure. So that's how we got around the need to like we don't have to we don't have to train the model. It doesn't have to learn information. We just have to set up the structure of the model, initialize it with with baseline reasonable parameters that we just we know from experience that these things that is going to work. We observe the behavior, make sure it's going well, and we can also kind of calibrate those parameters over time as we get data. But yeah, that's that's the kind of approach that that would take. And so yeah, we don't actually do machine learning in the conventional sense. We don't use large language models or or anything. It's very much kind of like almost like doing physics and and fitting having some intuition about like how parameters should be and then kind of doing some experiments that refine that intuition better. And yeah, so the specific type of AI like the expert system that that was more popular, I think in like the 80s and 90s. Yeah, it's not today. Today AI is kind of synonymous with large language models and neural nets and machine learning, but but it's actually there's there are other subfields of AI and we are doing it. And yeah, I think about it.
SPEAKER_02:  I think I've done the same thing. It's just you you're so good at something that you just made up. And then I called mine like heuristic based. Yeah, yeah. Then I've done it. Oh, the fuck I didn't even know that that was the name for it. That's super cool. But yeah, I thought initially I thought I was like how the fuck is it generating questions and like telling me all this shit. But if it's manual, then I get it. But yeah, that stuff takes so long. That's why I've actually got me into AI and like I have to learn to actually improve my own model. But I did have a side security. That was really different. Okay, interesting. So then there's no real training. So how do you test it and like find edge cases and you know, make sure it's actually working as you want it to work? Which super hard I did it. And that's not a fun process.
SPEAKER_01:  Yeah, for us, I mean, it's kind of come up. It's come down to starting with our one test student that who went through the that course and did well. And then just kind of progressively scaling up with more students watching, just paying very, very close attention to the kind of task. That they're being served, how well they're doing and quizzes having that AP CalcBC exam as our kind of our external validation metric. And also I have various validations that I've created to run against the database to make sure that things are logically consistent in the sense that like, okay, student is not student should not be getting a a lesson again, if they already passed the lesson. Unless they like failed it in the future. But like there's this logical thing like that there are things that we can check pretty easily for like, okay, this should not happen. This should not happen. This should happen. I have like hundreds of those kind of validation. So we just make sure that rails. Yeah.
SPEAKER_02:  And that like a certain trouble. Like I imagine the basic one being like, you know, you serve less than you do review. Yeah, like a week if they hail the review, give them a lesson again. That's like a basic condition.
SPEAKER_01:  Yeah. So it's various things like that. And, you know, so one of the just to give a concrete example of a way that we refine the model based on this is so. Yeah, we were always looking at students tasks and just hearing feedback from students and teacher. And so one of the things that we noticed early on, I think this was in, I want to say in 20 to 20, 20 to 21 school year. So we were looking at these student tasks and we had one student. I think I think they were doing like, I want to say like, pre calculus level stuff like integrated math three. It was either that or like one year algebra. And they had this task that was had to do with matrices. So they had learned like, okay, these are the elements of the matrix work. And then these are the matrices. These are the how you multiply matrices. And we noticed that they kept on like getting reviews on like the elements of a matrix, like identifying the element at a particular like, even after they had learned how to multiply matrices. And it was getting to the point where like we were looking at like this kid's task and we're like, this, this is dumb. Like a tutor would not like, okay, most things are working as as intended, but there's like too many easy reviews here. Like, there's just way to this is silly. A tutor is not going to go through like, okay, introductory linear algebra. You've learned how to manipulate matrices and stuff. Okay. Can you point me to an element of this matrix? Like, no, like the student already knows that. Why did they know that? Because they've been layering so much knowledge on top of it. And that was the moment like when we realized like, oh, we, we need to have encompassing in the database. We need to know what topics encompass other topics. And the bottle has to has to reason about that. It has to trickle space for petitions down the graph stuff. Like, so, and that came from just looking at a student's tasks. Jason was the one who really pointed it out and brought my attention to it. He was like, man, this, this kid is just wasting time on these reviews. But like, however much time, like five minutes spent on this review, like waste of time. That's silly. It's bad. And so we, I guess, yeah, came down to like you were saying earlier, earlier on, like, don't do the bad things. You want to do, you want to be a good? You want to be an expert? You want to, you want to perform? Well, just like, don't do the bad things. One of those bad things is giving students silly reviews that are so easy that they're just like, why am I doing this? And then they just ace it. So just do that. Yeah. Rinse it or p over and over and over again. When you get a very sophisticated system out of it.
SPEAKER_02:  That's interesting. I guess then for reviews, for like lessons, you would like put them in categories of like, oh, this is using this type of math. Let's say it's like using reciprocals and like complex numbers, for example, and then goes to differentiation. And then you can like cross out all of these reviews because it's all in one. And so you provide a different one, something similar to that.
SPEAKER_01:  It's a little similar to that. It's, it's more a fine grained though. It's actually so the way that we encode these encompasses is it is a an encompassing is a relationship between two topics. So is that one particular topic encompasses this other particular topic? But, but yeah, it's the same thing. We're like, yeah, if there is an encompassing, you can, you can knock out the review. And we also do like partial encompassing. So it's like, it's like part of this topic is encompassed, but not the whole thing. So you can count that as like maybe 25% of an encompassing. And so it doesn't fully knock out the review, but it just postpones it. It puts you a little bit further in the space repetition process.
SPEAKER_02:  So what happens if you don't ace the review, you get like maybe one right, fail one and then get the other two, right? So then you don't get the extra two XP, but you get, you know, what happens there? Is it like a module thing where you're giving specific review questions and combine them to make a review or?
SPEAKER_01:  So the review is the questions. The review are actually just selected randomly from the pool of questions that are like the last half of the lesson. So like the harder questions and the less, the ones that are more representative of the real thing that the lesson is scaffolding up to, not only the beginning of the lesson, but just, yeah, the more representative questions. But if you, yeah, so if you pass the review, but you don't get everything right, then basically what it does is it still puts you further in the space repetition process. This still counts as a repetition, but it doesn't give you as much credit as if you had aced the full review. So it's kind of like we do this kind of fuzzy sort of space repetition. Normally in space repetition, you think of it as like, OK, you do the lesson, and then you have review one and then our review two and then our review three. These are like discrete intervals. But with this like trickle down credit, partial encompassing and like partial accuracy on the task, it all kind of feeds into this fuzzy metric of like, OK, how much of a repetition was this performance worth? So if you just barely pass a review, you're not going to get a full repetition. Might be like a half a repetition or maybe even less. If you ace a review, then you're going to get more than a full repetition. You're going to get like maybe one and a half repetitions. And then there's also we also in addition to just this like specific moment in time performance, we also measure just the students like overall. Accuracy and how they're doing in general and use that to speed up or slow down the space repetition process. So it's kind of like there's the amount of credit you get for the for how well you performed on the topic. And then you multiply that by this this particular learning speed on that topic based on your overall accuracy. And then it's also we were also measuring how difficult topics are for students in general, some topics are just intrinsically easier than others. And so then we also have a factor like that multiplies. And so you get this like just every single spaced repetition is calibrated to how well you performed on the topic, how well you're doing overall and how hard that topic was and it's your learning speed on that. That's the mental that's the model to have it in mind when you think about the spaced repetition.
SPEAKER_02:  When I had when I was doing reviews, sometimes like it's actually quite easy and I quite fuck up because I read it wrong for some reason. So weird, like I see one number and they look back and it's actually a different number when I fail. I'm like, what the fuck? I've yeah, I do. It's all the time to me. So I'm like, Hey, so these questions are actually quite easy. And then like after I fail that and a new one comes up, it's actually significantly harder like what the fuck. So the failed review questions are like so much harder for me than the actual like initial ones and like damn.
SPEAKER_01:  Yeah. Yeah. So right. So our question selection during a task or during a review task is kind of just a random. So it's not that we're intentionally giving you a harder question. It's just the luck of the draw. So slightly the questions can vary slightly in difficulty. We actually, so we ran initially this was something that we looked into and because the question of the thing was like, should we work on a dynamic? Like, should we introduce something where if a student gets the question wrong, then we kind of try to select a slightly easier question. Of course, like that it's not going to be a lot easier because this is from a pool of questions that are pretty similar. But should we fine tune that? Is it worth the work needed to do that? And if they get it right, we use like this slightly harder question. Now, the thing that makes us a little tricky, I mean, it's stillable, but it's non trivial because at the same time we can't be lowering the bar for success. Yeah. So you can't if you if you can only do the very easiest problems in this pool of questions, then then that's just it's not. We have unintentionally lowered the bar for success. If we if we do that. Yeah.
SPEAKER_02:  It's really like we're a lot of way left.
SPEAKER_01:  Yeah. Yeah, exactly. Yeah. And so the benefit of the random selection just means like we don't have to worry about that. And so we ran this experiment that actually looked at like, OK, just based on when students go through through lessons, based on the random selection of questions, we have some instances where this kind of adaptive difficulty was just just by random chance, it was kind of emulated by the random choices. And we also have instances where it was not emulated by the random choices or the opposite was. And so the overall we we we we we we analyze the success rates in those two scenarios when the random behavior, the random question selection just happened to emulate the more nuanced difficulty adaptation versus not. And it turned out that it just it didn't really make a difference in terms of student passing the lesson, like the path rates were pretty much just identical. And so what that suggests is just like the difficulty, the level of difficulty variation is small relative to the knowledge that a student needs to acquire to to complete the question and that it was to get the question wrong that they still end up acquiring like enough knowledge from that experience to to be overall successful in future questions.
SPEAKER_02:  You know what would be really nice actually for the harder ones? Because I think there's no option to just like look over the past and just have some past ones.
SPEAKER_01:  Yeah, we've actually thought about something, I guess kind of in this similar spirit, but I mean, that's a it's a good thing to think about it as you've described it. But the way that we thought about something similar in the past was like, okay, for people who have kind of fallen off the wagon and they haven't done a whole lot of, they may maybe they skipped the previous couple days that they had scheduled to do XP. And they haven't done anything that's currently what happens is, yeah, you come back in the system and you have the same tasks as before. So it's like that the trader basically wants you to continue lifting the level of weights that you were doing, which is like, okay, if you're a personal trader, you're probably gonna, you're gonna say like, okay, why don't we do a little easier workout just to get you back on the train and make it the easier for them. And so, and we know, we know how difficult various topics are, because that's one of the metrics that we measure that goes into our adaptive space repetition and learning speed. And so, the idea was like, okay, well, if somebody has fallen off the wagon a little bit, maybe refresh their tasks and intentionally choose the easiest lessons available and tell them something like, Hey, if you just do one of these, then we'll give you like 50% XP boost for it. There's something like that. Yeah, you can get into the line and make like a happy path back to back to have it. Right. Right. Yeah. But I mean, what you mentioned, yeah, like a rapid fire, like review thing. Yeah, that'd be worth thinking about too.
SPEAKER_02:  Because I think it's a pretty similar business model with, I mean, you could say it, but like a lot of these things which require habits like the gym or like signing up for a we work or something, you know, they, oh, I think most of them actually bank make a lot of, a lot of money on the people that sign up and never do it. But then the people that want to come back, it's just so hard to like get back into it once you maybe something happened in life and you have to take a break and you have to come back, you know, like shit, well, it's so much harder. And then it's just back to like just paying the subscription and actually going. And I think a lot of people, this happens with all subscriptions though. So like having something that's, it's like, it's like on ramp people again, which is, you know, it's doing something easier to build momentum. And then you get like, yeah, it's just a snowball effect at the end of the day. And then that would be super helpful. Like I remember I think it was you or some other guy posted like this rapid fire times tables. Oh, yeah.
SPEAKER_01:  I think that was a mirage cut for from learn to be. Yeah. Yeah.
SPEAKER_02:  And I did that. I was like, oh, this is pretty fun actually. You know, and then you can notice which ones you take longer and you focus on that. And I was like, oh, this is actually pretty good. Like on ramping to like start the day with math, you do something easy, then it ramps up. You go do something on math Academy and you're in a flow state. Yeah. Yeah.
SPEAKER_01:  I like that quick five minute or five minutes of jump rope or whatever at the morning.
SPEAKER_02:  Yeah, exactly. So that's that was something I started doing. It was pretty good. And it's something actually up when I was listening to the other podcast with James that he did before. What was very interesting of like, you know, the automation of knowing like the fundamentals, like all your timetables, even arithmetic, some people don't do the experts. And I went down this rabbit hole of points and memorized my all my exponents. Cause I started to notice that I was memorizing and like all of these, like off by heart, you know, like four squared and then like six squared and all this shit and even even like cubed as well, I started memorizing cubes. And that was surprising because I had been doing it. But then like some I have like some times tables, which I don't memorize like perfectly like nine times eight or seven times. It's just like a very small amount of like those nine times ones. But like it's so annoying. And I was like, okay, well, maybe I got to do this every day of like read them or like do a lot of skipping. And for some reason, the exponents are much easier than the actual, the actual times tables of like nine times eight. I don't know why I can't remember it as well. But that was something. Yeah, that's interesting. But it's super important though. Cause as you said, if you don't, if you spend time thinking about the stuff, it like tells you from the path of creativity. And it's just like it's like writing in England. If you're trying to write like a paragraph and you try to figure out what punctuation is like a semicolon or, you know, a nastrix or any of this, then it's just going to slow your creative flow down. Like you distract your flow state basically. And that was super important. And yeah, I went down like this whole kind of thing and like, huh, maybe I should. And like I saw this thing with like John von Neumann. He knew like it was really crazy. I think he knew how to do division of like eight digits or like multiply eight digits. And I went down this rabbit hole of like mental, mental math of like, you know, you do calculations in your head. So using a calculator and I started to do like square roots and like fractions in my head. And I tried to not use the calculator. And that that was like a lot better at understanding, but it was super hard. I don't know. Did you do any of that before? Like try and memorize super high types tables and figure out like these mental metrics.
SPEAKER_01:  Yeah. Yeah. So I've looked. So I haven't spent a whole lot of time, but I definitely like probably like most people who are just interested in that. Yeah. I've gone down the rabbit hole of like how, how big of numbers can I multiply together? And yeah, it is kind of interesting, those kind of algorithms that are used. Yeah, it's all basically, yeah, but the limit is pressing up against your working memory and you have to come up with tricks to kind of compress some information together to open up more room.
SPEAKER_02:  Yeah. Cause I thought, I mean, like people thought like John von Neumann's success was partially attributed to like his ability to multiply, you know, six, eight digits. And it makes sense. But like, do you really need to be able to do that at the end of the day? So like, I don't know. And then I was like, Oh, maybe creativity is really just the only thing. And who cares about these like giant calculations? That's why you use a calculator for. But then if you go one step further on the argument, you're like, okay, why even do math? It hella limbs can do it, right? So yeah.
SPEAKER_01:  And I, so I think with von, yeah, I know, I mean, he, I wouldn't say that the multiplying of like huge high numbers was, was causal in his success. I think it was more of like, that the factors leading to his success also predisposed him to be interested in and capable of multiplying these numbers. But yeah, you can most professional mathematicians do not know how to do large multiple case.
SPEAKER_02:  It's more of like the actual reasoning behind everything. That's the most important.
SPEAKER_01:  Yeah, it's just the things that come up often enough. You need to know how to do quickly. So like times tables, those come up all the time, like your basic, like single digit, like within 12 or whatever, like multiplying by 10, those things come up all the time. So yeah, it's good to, it's important to be automatic on those. But like 456 times 872, like, yeah, that just like, you never really need to. Yeah, it's unnecessary. Right. So and like however long it would take you to do that, you would actually be able to do it faster with a calculator. Whereas, yeah, I guess I think that's the thing to keep in mind. Like, would it be, even if you were able to do this in your head, would it actually be faster to do it in your head or, or with the calculator? And for single digit time, it's like definitely in your head. It's huge speed boost. But once you once you get to like really long computations of big numbers, it's like you're, yeah, even the fastest mental computer would not have a chance against against a, a, a new electronic calculator.
SPEAKER_02:  I wonder like, what's the point of like, stopping, trying to automate all these calculates? I guess that's a really good point. But I think another important point is like, how common is the thing that pops up? And then, yeah, depending on that is, I mean, you should automate it. I know like you think about the square root of four or something. Then that's, I mean, that's pretty basic, but like understanding like logarithms. I didn't realize this, but like until I started doing that Academy, but like logarithms, exponents and square roots are so incredibly important. It's, I would say it's exactly the same importance of like multiplication, division, et cetera. And like same with reciprocals, like that's something I didn't really understand. And I wrote like an article on this on these basic things because I see that as like exactly the same as multiplication. And then being able to write that instead of using calculator for these things is super important. And yeah, it's just something I didn't realize until I started doing math Academy to be honest. Well, like math, seriously.
SPEAKER_01:  Totally, totally. It's one of those things that's like, it doesn't, you don't really realize the importance of it until you start doing like high level math and having to actually do the computations yourself and then you, then you, yeah, then you realize how important it is to, yeah, now to your logarithms are really well, your roots, et cetera.
SPEAKER_02:  And it was like, I was explaining it like in very simple terms, like understand why is important, why was it created. And, you know, just like, yeah, like what all the pieces of it mean, like the logarithm base and then, you know, your input, what it means and what you're, what's the use case of it? Like it's just a tool at the end of the day. And it's like in a language, just think of it as a symbol or character that is used very often. And then what is it? What's the underlying meaning of it? And why is it used? Like, why is the semicolon used, for example, when do you use it? That's, I think understanding this because this is eventually what builds up like very advanced fieroms. And it's just fear of the components of like very simple things at the other day. And so understand the simple things, how they interact with each other. That's how you get like that fluency in math. At least I think I don't actually have fluency in math. I'm just going from like how I would build it though.
SPEAKER_01:  I think the language analogy is very good. Yeah, it's seeing, especially in a comparison of the logarithm operation, comparing to like a particular kind of character or something or a word that just gets used often. And, you know, yeah, I think that's exactly the right way to, I think you're spot on with that.
SPEAKER_02:  If you can't understand it, you could never build these sentences or strings of multiple of them. And like, I mean, that's basically what like, you know, like co-variances as well, it's just like combining all these things together, covariance or under gradient descent. It's like, it's all the same shit, just like arranged in different ways to create a bigger idea. And then, yeah, if you're like another analogy, you could say like biology is the same shit, you know, single-cellular logarithms to multi-cellular is just compounding more of the same shit to build a more advanced thing. But yeah, that's my tattoo. Oh, I think it's good analogy.
SPEAKER_01:  Yeah, the math to like math. Yeah, it ultimately is kind of like a language.
SPEAKER_02:  Yeah. Yeah. It's just a time thing at the end of the day, how often do you use it? It's like living in Japan, Berlin, Japanese, you'll register to pick it up faster. So yeah. But man, it's been such a pleasure talking to you again, waiting for pot free.
SPEAKER_01:  Yeah, yeah, it was great. I had a lot of fun.
SPEAKER_02:  Yeah. Man, but until then, do you have anything to say for the aspiring math students how to get to a higher level, better level of math, mathematical understanding of the one C?
SPEAKER_01:  Yeah, advice for aspiring math students. I would, I guess I would say, yeah, in addition to just being strong on your foundations, if anyone is hoping to do a degree in math and is thinking that all they need is to learn like AP calculus. Any high schoolers out there who are thinking all you need to know is AP calculus and then you're used to being at the top of your math class. And it's going to be that way in college. Just learn the basics of proof writing to learn your set operations, learn, yeah, just get get some proof writing experience because otherwise it's going to be a rough ride. If you end up in a serious math program, I think there is a gap of missing knowledge that oftentimes this is not filled enough by whatever onboarding resources these serious math programs have. Sometimes they have like summer, summer work to like, hey, if you're to prep for your courses next semester, check out these these resources or whatever. But oftentimes it just it doesn't it doesn't it doesn't make up for not having a serious course. So yeah, I would say that's the one most important thing that I would recommend any aspiring that major is in addition to being consistent on your foundations, having reached level of automaticity, being able to read logarithms, roots to algebra quickly, just like it should feel just like reading a paragraph of text. Yeah. Absolutely get to that same level with basic proofs as well.
SPEAKER_02:  I think math is just a language to treat it as learning Japanese, do it every day, think about every day. I mean, you're just going to speak it and think about it every single day. It'll become fluent. It's just like when you're a child and you're around your parents, it's like they rush it in English. You load both languages because they're speaking at all the time. And so it's just like a build up over the decades basically and that's why you know your language so fluently. It's because you're just around it all the time. So I find the same concepts about it basically. That sounds head talk. All right, sick. That was beautiful. Thank you so much, Justin, for sharing your own experiences with your next videos. And we'll definitely have you on again down the road. Yeah, I'll be happy to. Hot 476. All right, man. Thanks for coming on again and we'll speak again soon. Goodbye.