SPEAKER_01:  All right, delighted to be joined here today by Justin Skysack, who is Chief Quant and Director of Analytics at Math Academy, an online adaptive math learning platform which I've been using for a round of months or so at this point. He's also the author of a wonderful work in progress book called The Math Academy Way, which is an absolute treasure trove of insights about math education, heavily references the literature on learning, deliberate practice, and talent development. I really recommend it to anyone interested in these topics. Welcome to Justin.
SPEAKER_04:  Thanks, James, Xander, it's great to meet you, brother.
SPEAKER_01:  Awesome, so Math Academy has become very popular on Twitter to the extent that people report that they're even quitting their jobs to use Math Academy. And I wanted to ask you, why is this the case? What's your analysis on it?
SPEAKER_04:  Yeah, well, I guess, yeah, I was also surprised to see some people quitting their jobs. I'll say that was not something that I expected, but I guess my guess, I mean, I can only speculate on it, but my guess is that some people have been wanting to make some career transitions into machine learning or just maybe tech in general or some quantitative field. And maybe my guess is that maybe this has kind of pushed them over the edge and feeling like it's really possible. If they just lean into skilling up, then what once seemed like an aspirational dream that was so far away is now within reach and all they need to do is level up their math skills and they'll stick to landing. But I was actually talking to one of these people yesterday. Like last night, we had like a three-hour conversation, actually. It was crazy because I saw like, he said that he was quitting his job as a teacher. And I was like, dang, that's a big career. But this is like right at the beginning of the year. And I knew he was going to be making a transition to software engineering that he had on deck. But I guess in addition to the whole math academy thing, he was just getting a little fed up with the situation this year. And then he said, I'm not sure if he had a class of like 53 students, average class size of 40. Like that's just ridiculous. So yeah, so I guess like I'm sure various people have various reasons. But in addition to the like yeah, aspirational like learn math, get in a career situation you like. There's also this like teaching nowadays, it's kind of a pressure cooker for a lot of public teachers, or it's just stress on top of stress on top of stress. And eventually like there's this straw that breaks the camel's back. So it's like it's a stick and carrot situation going on.
SPEAKER_00:  Right. So since you mentioned the teachers, have you noticed that there's a certain category of job that makes someone more likely to want to make this transition and learn math and math academy and get a job there?
SPEAKER_04:  So I would, I mean the two that pop out at me is, there's, yes, so this teacher was a math teacher. And so there's a lot of math teachers who we were actually talking about this last night. They go into the profession of math teaching thinking that it's going to be like they're just going to have fun learning new math, exploring math with kids. And it ends up being a lot more administrative burden and just like making sure you have to kind of become a task master and make sure kids are on track. Because there's like maybe in your class of 53 students there's like five who actually aren't really interested in the material and they're trying hard. And then there's like 20 who are like, they don't hate it, but they're not particularly motivated. And they're just trying to get them aligned a little bit better and motivated. And then like the other half is just like they're just kind of against you. Like they just don't want to do any work. You have to basically be the drill sergeant and just say like, no, I'm going to hold you accountable. And like that's, if you go into the teaching profession kind of just hoping to have like you know a dead poet society sort of like very inspirational, bring out the creativity and everybody like it can. It's just it's not always the most fun thing to have to be a basically just be a yeah, a drill sergeant to some of these right now these kids. But so yeah, that would definitely influence like I imagine a lot of math teachers are in a similar position. But I know there's also quite a few software engineers who I don't know if they like I guess I think one of them left this job, but I think in software engineering it's a little easier to kind of like keep your keep things the way they are professionally and then just kind of do a a like try to take on more machine learning projects or math. Right. So if you're already in the arena, you don't have to like duck out and join somewhere else. It's like, you can kind of just just kind of fly your way into it.
SPEAKER_00:  I mean, that's that's James's situation. Yeah, he's already a software engineer. I mean, I don't know when you're job explicitly to do math academy only, but certainly that's been a huge portion of your time.
SPEAKER_04:  James tell me about that all of it.
SPEAKER_01:  Yeah, it was interesting. So I learned of math academy a while ago because I was working at a company called RemNote, which is a space repetition company. So they're basically working to create sort of a fusion between notion, the note taking app and Anki with the flashcards. Yeah, I've heard a book. And I saw your blog post on the on the space repetition algorithm you'd created for math. And I posted it in our Slack channel because we do a reading group every Friday. It's like, we've got to read this paper. We've been talking about, you know, similar ideas about how we can optimize space repetition using actual semantic details about what's in the flashcards, right? And then I kind of forgot about math academy for a while. And then after I quit my job, I was on Twitter, made some posts about how I quit and what I was looking to do going forward. And a lot of people suggested to me to check out math academy. And when I checked it out, I obviously remembered that you'd written that blog post and I was like, oh, okay, I've definitely got to check this out. And yeah, it seems really awesome.
SPEAKER_04:  Yeah, that's great. Yeah.
SPEAKER_01:  But yeah, the motivations were definitely basically to work on things that are more technical and more, I guess, interesting, interestingness is subjective. But it's always felt to me like I've been blocked or hampered in a sense by not having any sort of math foundation, because every time I want to do something in machine learning or scientific, there's inevitably a certain level of mathematical maturity that you need that I was just completely lacking. And it made getting involved in those things very frustrating to the point where, eventually, if I want to get into those things, I need to take it seriously. It needs to be my number one priority. So I kind of just have to take the leap, you know?
SPEAKER_04:  Yeah. Yeah. So for you, it's like you just want to focus 100%. I'm taking this step landing. Yeah. The transit. Yeah. Yeah.
SPEAKER_00:  That makes sense. Yeah. It's a good thing because you mentioned the, oh, sorry, just real quick. I just mentioned it. You mentioned the teachers having to motivate their students. I imagine for most math academy students, it's on the complete opposite end. These are very highly motivated people. It seems to require a lot of motivation to get through it just because of the volume of content. Not to say it's inefficient, obviously. It's extremely efficient, but there's still a lot of effort. And you guys, it's like a, it's an effort forward kind of marketing. It seems to me with, with the mentions to athletes, which I completely love. I'm, I'm very, very interested in that, that way of doing it. It's very targeted. They say explicitly on the site, you know, this is not, this is not meant to be easy. We're training athletes. It's like, it's like a kind of athletic training, but for mathematics. And so I like that. It seems to me that most of students like James, like James and the math teacher or anybody else would be very likely to be highly motivated on their own.
SPEAKER_04:  Yeah. Yeah. I would say, I would agree that being successful on math academy, yeah, it requires some level of motivation. Whether that's intrinsic motivation to learn the material or extrinsic motivation sometimes, like maybe you don't like really like math or math's sake, but you really like machine learning and you know that this is just like you got to eat your veggies to get them getting strong. So, but yeah, it's 100% true that if somebody, if somebody comes in thinking like, like, oh, math academy is like, they say that they're like super efficient. That means I don't really have to work very hard at all. And then they take that attitude. Like that's, no, that doesn't work. That's not, that's not the way we mean efficient. Like you say, it's like, it's, it's, it's very, the training is pretty taxing. That's where the efficiency comes in. It's like, I guess the way that I usually try to describe it to people is like, okay, like think about if you're signing up for a gym and lots of people will go in to the gym and they'll just start doing some random exercises on random machines and like in between sets, they'll, they'll, they'll do like a couple reps on the, on the bench press, then go sit down for like five minutes, play on their phone, text friend or whatever. And like, they'll do that for, for like an hour and call it a workout. But it's like, that's very, very low efficiency. Right. When we say the way we get the fish inside of you, it's, it's kind of like imagine you go to the gym and instead of just, just being kind of let loose like, hey, do, do whatever you want. However much you want sweat, don't sweat. I go, okay. No, it's like there's, there's some, um, ex-alliptic sprinter that's just standing there waiting for you to be like, Hey, you, you want to work today? And then you, you got to go and say like, yeah, yeah, I want to, you, you tell, you got me for an hour today. I'm going to do whatever you tell me, like use whatever training methods are, are going to, are going to move the needle most on my strength and my speed and everything. And then like the sprinter is, is going to like, they're going to make you work. You're going to be sweating a ton. You're going to, you're going to feel completely wiped out afterwards. But what they're not going to ask you to do anything that you're not capable of doing, everything is going to be within your zone of capability, but you are going to be working hard the whole time. And that's, that's where the effort comes. And it's like, you pack the max learning into the, into this, this amount of time. And it's like, you feel it, you feel it. It's like you're, you're doing these like circuit training, super sets and the workout. You're like, you just do an exercise. Is it rest time? No, it's time to move on to this other thing that we got for you. You're, you're possibly going around making the most of your time.
SPEAKER_00:  I completely love that man. That gets me so excited. And, and you know, we all know those people just to bring it back to the gym analogy. We all know those people who've been working out for a year or two years. They look the same as they did before. They're not in any better shape. And you go, what do you keep going to the gym for? All the time you say, I got to go to the gym. And it's like, you look the same, what's going on? And there are some people, three months later, they look way different. It's like math academy, it seems to me, is that equivalent. Three months later, your, your mental space for mathematics is totally different than what it was before. And that's what you're trying to get. You're not, not trying to dilly-dally around, make a little bit of progress and step back and, and repeat it again over and over. You're actually trying to make progress and build on.
SPEAKER_03:  So, I love that.
SPEAKER_00:  It gets me honestly so excited. Like, I just feel completely amped up listening to that.
SPEAKER_01:  It's very funny for me watching him talk about that, because I can just see in his eyes that you were saying exactly what he wants to hear. Just saying, you couldn't have said so.
SPEAKER_04:  I love it. I love it, man. That's the kind of people that we want using math academy. There have been people who have asked me like, hey, I only want to do like five minutes of math a day. Or like, I want to, can I do a little, just flip through my phone like a couple minutes here and there. Like, math academy is going to help me do that, right? Because it's super efficient. And I'm just like, no, don't. We don't, you're not the kind of person who's going to benefit. That's a completely different optimization problem. Like, a lot of people, they, they want to like, learn some like superficial level of math to like, very surface level. Not, you, you're not comprehensive. You can't really like go code something up with the math you learned. But hey, it's kind of kind of cool. You, you, you learned like the idea of integration is like can be interpreted as area under the curve. Whatever. I mean, that's cool. But no, we are, we are covering the full standard. You, you're going to learn how to do everything. And we're, we're, we're not trying to, we're not trying to optimize for a small amount of time. Right. You are coming in like you would with a workout like at least a couple hours per week total. And you are going to pack the most. But yeah, we're not, we're not afraid to, to, to turn people away. So. They are not the ideal customer.
SPEAKER_00:  Yeah. Since we're on that, that point of time, you mentioned five minutes a day is too little, which that's obvious to me. How much would you say is like the lower bound where you're actually going to see good results and still make faster progress than you would just kind of dilly diling around. And then just out of my own interest, and I'm just curious if you have any sense of this, is there's some kind of upper bound where you're trying to shove too much mathematics in your mind. Like, can you do six hours a day of it? And I literally mean that. I mean, six hours of focus effort. And even if you could, does it work the same way? Or is it one of those cognitive things that truly has to be spaced out a little bit more than that? Just kind of the upper and lower bounds.
SPEAKER_04:  Yeah, that's a great question. Oh. So as for, as for lower bounds. Yeah, I, I typically say like the cutoff is like a couple hours a week, or like maybe half an hour a day. I guess like 20 minutes a day, maybe would be, well, once you get started going lower than the 20 minutes, then it gets to be like, what, what are you even doing? Are you even, are you even here for your workout or, or what? Right. Now the, the one exception to that rule, I think is younger kids. Like if you have a, like a, like an eight year old or, or something using, using this, like they, they just kind of like limited attention span.
SPEAKER_00:  And so if somebody. Just look quick as a point of contact for people who don't know, and you can tell me if I'm getting this right, is it correct to say that math academy lessons are about 10 minutes or they're abouts or without, is that it incorrectly?
SPEAKER_04:  It kind of, that's about, all right. It kind of depends on the particular topic and the particular level of math as you get up in the higher levels. Just, you know, things get more complex. Sometimes it takes longer. Sometimes you get like 20 minute lesson or so. But for the, for the lower grades, yeah, 10 minutes is kind of like a, like an upper, an upper bound typically for the lower grades.
SPEAKER_00:  So at least a couple lessons a day. Okay.
SPEAKER_04:  Yeah. Yeah. Yeah. So in fourth and fifth grade, like the courses are relatively small compared to, say like, AP calculus or, or like linear algebra or something. Like, so you can still, like in terms of like what students are expected to learn at going to the school system in grade levels, like even something like 10, 15 minutes a day in one of those lower grades will, will, will get a kid moving pretty, pretty quickly. So once you get up into like more, more serious levels of math, then you kind of have to put a little bit more, uh, force to hide your efforts.
SPEAKER_01:  I see. I'm curious to know, um, based on your like survey of the literature, we've been talking about sort of, uh, analogies between, uh, going to the gym and, uh, doing mathematics. You know, what are the most important insights that you've gleaned from the literature on, like, uh, what we can learn from talent development in like music and sports compared to mathematics. And how are you like building that into, uh, math academy?
SPEAKER_04:  Yeah. Yeah. That's a really good question. Um, let me, uh, actually let me first address the other part of Xander's question, which was like a high end. Uh, like, so if, if somebody puts in, uh, more, had a couple hours a week, like, that's great. Um, there's no, like, um, it's not like the learning process breaks down when you're putting extremely high, high efforts in there. And it's actually, it's, it's, I would say, actually jams the tide into your question too. It's like, it is essentially, it's, it's, it's just like, uh, how you'd expect with, uh, somebody who is training for serious athletics. Um, these, these people who are very seriously training can go for, for multiple hours a day in their training. Uh, now there, there is like a point where, like, the train, I mean, the, the, the deliberate practice, the, the, just being on all the time focused doing things at the edge of your ability. It's very taxing. And there's a limit to, to kind of how, how long you can actually do that productively, but everybody kind of has a different, a different breaking point in that. Um, and yes, for some people, it's like a one hour day and there's, they're just totally winded, like, they, uh, it's like they have the, the mental equipment, like, um, just, like, being, uh, just like the arms and legs feeling like they're gonna fall off or something. And it's like, okay, it's time to take a rest, come back the next day. But, um, some other people can, can go for longer. And you can also kind of like train this up to your, your stamina. That's the thing, again, in athletics, it's, it's like, part of, part of the, the benefits to just being in shape and having consistency in your training is that you have the stamina to do longer training sessions. Um, and so it kind of builds into itself. Like you can imagine somebody who's like, you know, the, maybe on the off season, um, some serious athletes get a little out of shape and not really exercising that much. Not really eating the best food. And they come back to the, to the regular season and the first day of practice just kicks their butt. And like, they just, it has to be maybe a little bit lighter, a little bit easier than they get back into the practice. So it is, yeah. Um, but nothing, nothing breaks down as you're, as you're going through, through higher, or like, going through, um, lots and lots of training each day. Uh, now, the only, the only way that it was really break down is, is if you decide like, hey, I'm gonna do, like, uh, ten hours of work, um, and then, uh, you know, you're gonna be able to do it. Like, uh, ten hours of work on one day, and I'm gonna wait like four weeks of work on the back. Right. Because then you're kind of like forgetting a lot of what you, what you did. Uh, but if you, if you are, if you, if you are consistent, you're doing like at least a session every, every other day, say, like, several times a week, and you're putting up, like, serious, uh, a lot of time, a lot of earning a lot of XT. Then, then, no, nothing breaks down. You, you just, you just move faster. That's, that's pretty much it.
SPEAKER_00:  I say, so, just to put a fine, point it on, fine, point on it, I guess, there's no, I guess you'd say equivalent of overtraining, even if you're doing six hours a day, you're still gonna retain the information and learn as the system would intend. Uh, yeah. If you're able to sustain that focus, intrancy.
SPEAKER_04:  Yeah, yeah, exactly. I, I, I would say so. Now, I guess I'm a little hesitant to say there's no such thing as, as overtraining. Uh, but I, sorry to phrase it that way. No, that's okay. That's okay. But I would say as, yeah, as long as you don't. Okay, what I'm getting is like, when you, when you overtrain athletically, like you can, you can feel it. Like you just, you know that you're not operating as, as well as you were. Uh, initially, like you're, you're, you're, you're encountering progress problems, the needles not moving. You're just feeling really fatigued and tired. And, and, and so I would say as long as you're not feeling that and you're learning, I guess, I guess in learning. Uh, the, the, the analogies is like burn out. If you're feeling like burned out and like you're losing motivation to do the thing that you're, that you're, that you're learning that. Then, yeah, it's time to, time to tone things down, take it a little lighter, give yourself some more recovery time. But, but if you're feeling great the whole way through and you are making, uh, progress, like you're, you're doing well, you're passing, you're, you're learning tasks, you're feeling good. Like there's no, there's no reason that you should like cut yourself short if you wanted to do more. Uh, but, uh, oh yeah, at the same time you don't want to be the person who, who, who joins a, a gym then our first date is like four hours and it's so sore that they just, they never can actually. Yeah, exactly. So you do have to kind of pace stuff. But yeah, I guess it's, yeah, you gotta be like very, just kind of in touch with how you're, how you're feeling and monitoring your, your, yeah, your, your science there. Right. So yeah, I guess, so it's the, to jump to James question that James, your question was about like, um, it was about the principles of, of effective training. Can you just summarize again what, what that question was?
SPEAKER_01:  Of course. Yeah. So basically what insights have you gleaned from talent development in, like, music and sports, the kind of stuff that tends to crop up in the literature that you've been reviewing your book.
SPEAKER_04:  What sort of insights have you gleaned and how are you building it into the academy? Oh, yeah. Yeah, totally. Um, I'd say, okay, but one of the biggest insights is just the importance of having a strong knowledge of your foundational skills. You need to be automatic, but the key word is automaticity on lower level skills. And so the, the idea is like, um, if you're, I always make the analogy that like, if you're a basketball player, you need to be automatic on your dribbling. You can't, you're, you're, you're just not going to be able to function and do anything, uh, any advanced maneuvers. If you have to consciously think about like dribbling the ball, you're, you're going to, your head's going to be down. You're going to be looking at the ball. Uh, you're going to be like tripping over your feet. You don't know where your teammates are. You don't even know where the hoop is like, might just be running the wrong way. But you have to, there's a lot of, you have to be able to look up, not think about dribbling and, and kind of do a bunch of other things in parallel. And the same thing happens with, uh, with learning, uh, especially in a very hierarchical knowledge domains like math. You can, I mean, so imagine like, okay, analogy dribbling is like time stables. Uh, if you are trying to do algebra and you don't know your time stables, it's like you're, you're trying to do best. But without knowing how to dribble and right. And so if you want to, like, not only is it going to take you forever to get through as a single problem, but you're, you're, you're, you're, you're, you're, you're going to be just continually interrupted. Uh, cause you're, you're thinking, you're having to go back and think about these, these low level skills. Um, so. The particular example that I, that I often like to give is, is, is imagine, imagine a student needs to come is learning what an exponent is, and then needs to compute four to third power. That's best today's lesson. We're cubing numbers. So first, first problem, four to the third power. Um, that's just four times four times four. If somebody is automatic on their, uh, times tables, um, like, like, like a basketball play, who's automatic on dribbling, they can just think like, Oh, well, four times four times four. Well, just do the two fours initially. That's 16. Just pull that straight from every, not even thinking about this. Oh, that's 16, not 16 times four. Well, that's 10 times four, six times four at the results. 40, 24, 64, easy. What, like, what's, what's the big deal? It's like, um, yeah, you see, um, but if you, if you don't know how to do these, uh, these times table, if you, if you have to consciously think about, uh, like, recomputing the times every time you start to get lost in the weasel, they go like, okay, four times four. What is that? I don't know what that is. I guess I need to count it up. Let's see. Uh, four plus four, eight, four is 12, four is 16, and now do 16 times four. Like, oh boy, now I do like 16 plus 16 plus 16, and then maybe you get a result of like, 62 or something because you, you can discount it somewhere. And then, and then you like go and you're like, I got the answer. It's 62. And then Jesus is like, that's close, but not quite. And you're like, what? Really? I need to go through this whole process again. Um, and so like, and now you're like, you're not getting additional reps of practice because you're still just struggling on this same rep. You're not, um, you're not like, you're, you're, you're gonna miss out on little key things. Like somebody who can compute cubes very, very quickly. They're gonna go through a number of reps and they might see something like, oh, hey, every time I cube an even number, it comes out even every time I cube an odd number. It comes out. That's interesting. Um, because they're, they're like the input and output to their procedure is so, uh, it's so close and tight, but they just kind of notice, like, just notice these details, these things, but, but yeah, somebody's taking like, like, five minutes to work through a problem. And then you often don't really notice those trends and just, just take this stuff further. It's like, times tables being automatic on your times tables. Uh, it, it, it, um, that's, you can actually go step below that. Some students, like, not only are they not automatic on those times tables, but they're not even automatic under addition. They have to finger count. And you can, you can just imagine how this blows up when you have to compute something like four to the third. It's like, well, back to like four times four times four. Well, what's four times four? That's, that's four plus four plus four. But what's, what's four plus four? Four, five, six, seven, eight. I actually like, I made some, uh, in, in my book, the math academy way, I have a section where I like, I detail all the individual computations that are going on here. And like, it takes like a full page of just like how to compute four to the third. If you don't know your addition facts, by, by heart, you have to, to, to finger count that. Whereas, right, if you're automatic on your skills, um, that's like two sentences of thinking, essentially. That's it.
SPEAKER_00:  That, that, it's an interesting thing because there's, there really are some things in life where if you do them slower, you get better results because you're thinking more deeply. But there's some things where if you do them slow, you almost can't do it at all. Like, if you're a very, very slow reader, you can't keep your train of thought about what the paragraph was about. It's taking you so long to get through the paragraph that you actually almost can't read if you're reading too slowly. And it's the same thing here. So it's like, this is in that category of things where you do want to automate it away.
SPEAKER_04:  Yeah, yeah, that's a great point about reading. It's emphasizing up. Yeah, so there's, there's also, there's so many other principles besides automaticity. There's also just the idea of, of interleaving mixing up your practice. And so if you're a basketball player and you're trading, that might mean like, hey, don't just practice 50 jump shots from, from the same location in the court, taking the same stance. Like go move to other areas of the court, go throw in a layup, try to do a jump shot while coming at it from a different angle or getting a pass from a different direction. It's, so the same thing kind of happens in math problems is, or just in learning in general, if you give a student a task and the first time they complete the task, they, they have to load up some information into their mental RAM, into their working memory to, in order to complete the task. And the next time, if you just give them the same task afterwards, well that information is still sitting there. They don't actually have to pull it from memory. So what ends up happening is that the task is artificially easy because they've already pulled the information from memory. So they're not actually getting a full rep on it. They're just kind of mindlessly going through the motions. So that, yeah, that makes the task artificially easy, which at the beginning, at the beginning when a student is first learning how to, how to do a skill, like being artificially easy is, is good actually because it's like you're using a, a weight that is more calibrated to their, their level at which they can lift. But as they go through the learning process, you want to kind of strip that easiness away and have them pulling information from scratch because it's like, the way, the way that you, you really write something to your memory is having to pull it from scratch every time. If you already have the context loaded up in your head and you already know what you're, like you just, you just have to do the, the final part of the problem. Each time you already know, it's exactly like the previous problem that you did, then it's kind of like you're, you're doing, you're going to the gym, you're working out, but your spotter is like helping you lift the weight every time. It's like you have to go through the full, the full process of, of taking information from long term memory and physically lifting it up into working memory. And if you are just letting it sit in working memory and just leveraging it there, it's like you're, you're, you're, you're skirting around the fact that you have to be lifting this weight. You're getting your, somebody, you think you're putting up weight, but you're not actually lifting it. You're just letting it stay there.
SPEAKER_00:  Yeah. That's actually a really, that's a really interesting framing of why interleaving has a positive effect, just preloading into the working memory, which removes the benefit of active recall. I hadn't thought of it that way. That's, that's, yeah. Right. Would you say that counts for all of the benefit? I wouldn't think it accounts for all the benefit of interleaving.
SPEAKER_04:  Yeah. There's, there's more to, I mean, there's a lot of these learning strategies. They, they have, yes, so many different benefits. So interleaving, yeah. One of them is the fact that you have to load information from scratch every time. Right. But another one is that you also, you, it helps you train your ability to match solutions to problems. Like, for instance, like, suppose you, you give somebody a linear equation to solve and they solve like a bunch of these linear equations in a row. And they, so they, they're, they're, they've gone to the flow of this. Okay. Now you give them a quadratic equation. So it's, it's just a simple quadratic equation where they have to like, it's like x squared plus four equals equals, or x squared minus four equals, or something like that. They can just move to the scientific square root. So they, now they do some of these, uh, quadratic equations and they get into the flow of that. And now, and now you, you ask them a, a, the next day you give them an equation and they can't remember like what solution procedure goes with, with what type of equation because they haven't practiced actually distinguishing the skill. You just gave them a bunch of skill type A and have them execute solution A and they just knew that's what they were doing the whole time. They're like, okay, cool. Sounds good. We're doing a here. And then you gave them a skill type B and you had them execute solution type E. And yeah, they just did this over and over and over. But they haven't actually practiced distinguishing like part of the problem is like, should I use solution type A or solution type B? Is this square root of the problem? So the interleaving that helps you, like you, it makes it so that part of the problem is actually deciding how to solve it.
SPEAKER_01:  So it seems very related to something you posted on Twitter about interference between certain multiplications. People having memory interference, then struggling to remember which, you know, which calculation map to which answer is very interesting.
SPEAKER_04:  Yeah, right. Yeah. Right. The associative interference where it's like, if you have something like, I think it was three times eight makes 24 and also four times six makes 24 and you kind of, you just get used to like, well, okay, there's, there's an eight fact that maps to 24 and there's, there's also four fact that maps to 24. So when you, when you get asked, like, what's four times eight, you've got some interference from those other facts. Then you're thinking like, okay, well, there's a four fact and there's an eight fact. Both of those, I know other facts, but that's 24. So this pushes you to say something like four times eight is 24. That, that shows up a lot. Right. In multiplication mistakes. It's actually one of the top. It was weird because it's like you would not expect it to be one of the top missed multiplication facts, but yeah, four times eight is one of the harder ones. But yeah, it's like, so initially, initially when you, when you first teaching somebody, yeah, you want to keep these things separate. You don't want to interleave them together. You want them to just get in the hang of doing executing the skill in isolation without any other, other difficulty. Just make sure they can get over that hump. But once they're able to get over that hump of executing skills in isolation, then yeah, it's time to combine them and try to, try to untangle these, these are responses in their head.
SPEAKER_00:  So just to bring in another effective learning principle that's exploded, I guess, by, by math. I mean, do you think that there's an interplay between interleaving and layering? It would seem to me that that matching a solution to a problem is a kind of layering unless I'm thinking about in the wrong way.
SPEAKER_04:  Yeah, so I would say there was a couple ways to interpret layering. The way that I usually talk about it is layering on actual content knowledge. So it would be like learning more difficult techniques and just learning higher levels of math. But I think, I think, yeah, what we are getting at when you mention layering just now in terms of like adding some kind of difficulty to the learning process. And in terms of like distinguishing between solution techniques, the official name for something like that would be called desirable difficulty. So that's a term in the literature. And so the idea is there are various features that you can add to a learning task to make the recall process a little bit harder, but in a way that is productive for the learner. And so interleaving, mixing things up is one of them because the learner has to pull this information from scratch. And they also have to match the appropriate information to the problem that they're faced with. Another type of desirable difficulty is actually spacing out practice. So the desirable difficulty that that creates is when you space out your practice, you are letting your memory get a little fuzzy to the next cycle of practice. And so it is, so it's kind of like you're letting your memory get fuzzy intentionally because you're going to then overcome the difficulty to recall that fuzziness. And then by over by recalling the fuzzy memory, that is what actually makes the memory trace stronger. It's like your body adapts to like, okay, this memory was not as easy a recall state as we need. So we need to increase the retention of it a little bit more.
SPEAKER_00:  And for users of spatial petition, you can tell me if I'm recalling this correctly. My understanding is that the harder it is to recall something at review time, the more of a benefit it has to the storage of the memory. And so that's a big counterintuitive, but that's the principle of spatial petition. That's why you can have expanding intervals. Otherwise, as far as I can tell, it wouldn't work.
SPEAKER_04:  Yeah, that's exactly right. So you're upping the level of difficulty each time. I always think about this as putting weight on the bar in the gym, right? Like maybe you start out with like 100 pounds. It's pretty tricky to lift. But as you go to the gym and do this repeatedly, you need to make it more difficult for yourself. So you don't get stronger by just lifting the 100 pounds like onwards and definitely like you actually have to keep layering weight on the bar. And the context of spaced repetition, the weight that you're putting on the bar is, well, it's just how long you're waiting until the next recall. Yeah, so spaced repetition is like weight lifting, W-A-I-T. But yeah, so this is all subject to the fact that you're overcoming the difficulty. That's the one thing about desirable difficulties that people sometimes mess up as they think like, oh, well, well, all the difficulties are desirable, right? So why don't we just like throw all the difficulties at the learner all at once. We're going to have information be really fuzzy and we're also going to like jump straight into interleaving. The hardest kind of interleaving, we're going to throw all these tricky things at you. And basically the equivalent, what that means in a weightlifting setting, what that's doing is essentially taking somebody who just signed up for the gym and they're just learning to squat. And then you put like 500 pounds on the bar, you're like, that's what this. They just get like crushed by it and it doesn't make them stronger. It just like, they're not able to lift the weight, they're not putting on any muscle. They're getting demotivated too, because they're just feeling really bad by it. And yeah, and so the key to desirable difficulties is like, you want to make it as difficult as possible subject to the constraint that the learner is able to overcome those difficulties. So initially, yeah, that starts out with like all these things like waiting for the information to get fuzzy and interleaving, mixing the problem types up. These are not good to do at the beginning. At the beginning, you want to make it like relatively easy for the form. Even if that means artificially easy, you just, you need them to get to a point where they're able to actually overcome the difficulty. But then, yeah, you start ramping it up.
SPEAKER_01:  One of the things you've talked about is how all of these sort of learning optimizations, the sort of working memory optimization, the automaticity, that all kind of forms the prerequisites to become creative in using math later down the line. Could you talk a little bit about why that's the case and why people should focus on these sort of more drill style questions instead of the ones that are sort of targeting more creativity, like coming up with a unique solution to approve that would take you, let's say an hour instead of just a minute, like most of the questions on math academy?
SPEAKER_04:  Yeah, yeah, totally. Yeah, there's a lot to this. So I guess the first thing is like a lot of people think that this kind of drill style questions is like the opposite of creativity. And I think like, oh, well, you're just, they might think that like automaticity is the opposite of creativity, because automaticity is kind of like just doing things without even thinking about them, getting to the point that you're so comfortable that you can effectively be a robot doing those low levels skills. So they're like, well, wait, robots aren't creative, humans are creative, you want to not be a robot, right? So this is like, why are we doing this? But the idea is that you want, the part of your mental process that is being creative, you want to give it enough room in your working memory to do that. So that's kind of you want to be robotic on these low level skills so that they so that so that you don't have to exhaust your working memory on. Everyone has a limited capacity of how many, how many pieces of information, how much, how much just mental effort they can devote to solving a problem at once. And if you're spending all that effort on low level skills, like, like, like recomputing your multiplication facts or like your basketball player thinking about juggling, you're going to overwhelm your working memory and you're not going to have any room to think about high level features of problem solving or like look for your teammates on the field or think about yeah, any sort of like creative insights that you have to have to solve problem. So, so yeah, you do want to have like your low level skills. You want to be a robot on those to free up your kind of like human creativity for the higher level thinking.
SPEAKER_00:  Can I before you, so just so I could feel of my intuition is right here, I always think of this like, if you have painter and you're trying to paint something creative, if you don't have the technical skill of painting, if you can't paint what you see or paint what you think, you can never hope to be creative about painting because you're trying to figure out how do I even paint. You can't think of an interest in creative painting. You can't be Picasso without having the deep technical ability to just paint what you see or paint a thought that comes into your mind. It's like you have to master that so that you can be creative. Would you say that's a reasonable analogy? Because I always think of that.
SPEAKER_04:  Yeah, yeah, that's that's a great point. That's a great point. Yeah, I like the right because that's that's a real people like you say creative like yeah our arts arts professions kind of come to mind. Yeah, I'd say the same thing. Yeah, that's really good one. Yeah, you can apply that to music too. If you don't know like your your your scales and your your you don't have the basics masters of your of your instrument like you can't bring whatever creative thoughts you have into reality. It's hard to even think in a truly creative sense if you're just struggling to you don't even know the basics. Same in writing as well. Yeah, 100% right. Yeah, if you're not if you don't have a large vocabulary and you're not good with grammar and you're constantly having to like don't you can't have words flowing. There's just no way to have words flow if you if you're constantly having to think about how to. Yeah, the low level grammatical semantic details of things under personal degree. I think I accidentally interrupted you there if you were trying to finish up another point. Sorry. Oh, yeah. Yeah, no worries. No worries. Right. Right. We were talking about automaticity and rope.
SPEAKER_01:  Yeah, like kind of how automaticity allows you to free up your working memory so that you can be creative. Yeah, right. Yeah, exactly.
SPEAKER_04:  Yeah, I thought there was something that let's just move on, but we covered it.
SPEAKER_01:  So maybe we can move on to a slightly different subject, which is one of the things people are quite concerned about nowadays is that with the rise of AI, there's going to be no more need to sort of learn mathematics. Like I'm in a lot of different learning. Yeah, or anything for that matter. Really. I've been a lot of learning discourse and there's definitely like a large contingent of people who feel very pessimistic about the future and they might be studying by themselves for a while, but they see all these advances in AI and they think what's the point of continuing to study. So how would you put their minds at ease? What sort of advice would you give to them?
SPEAKER_04:  Yeah, right. So the question is these people who are, yeah, they're worried about like, what is the utility of learning math anymore and it's like is an AI just going to take over everything. Yeah, yeah, yeah. Yeah. So, yeah, you know, I, I gotta say that the perspective there, I think it just doesn't even make an intuitive sense to me, an any intuitive level. I'm sure it's like the same for you, but it's like anyone, anyone who's like really had to solve hard software problems that are not just like this HTML page together or like solve this lead code problem for me. Those kind of problems are just like very packaged up, very well defined. A lot of the problems in software are just like how to, well, once you get to a high enough level, it's like part of the problem is even defining what the heck the problem even is. Like how, like you have this, this, this, your software has to make life easier for somebody in some way, but, but it's like, but yeah, so much of the problem is just figuring out like what, what even the software is supposed to do. Once you know what it's supposed to do, then, then it's, it's a lot, it's a lot easier to actually actually write it, but, and so it's, yeah, I think at least today's AI tools are quite a long way from, from actually solving problems like that. I mean, that'd be great. I would love if I could just, if I could just tell like, like spin up one of these, one of these chat bots feed it like the math academy code base and just be like, hey, I want you to like calibrate all the space repetition mechanics in the model. And then like, it'd be great if it could, if it could do that, but, but so much of this code is like, it doesn't exist anywhere else online. It's new algorithms, new perspectives, like it's never really trained on these kinds of problems before. And like, what does that even mean calibrating the space repetition model? How do you calibrate it with the, with the particular data that's given? It depends a lot on what data is available. Another thing like, just, how do you, how do you improve the, the onboarding experience for, for users? Like part of that is just knowing what are the, what are the problems that, that students are facing that are preventing them from, from, from using the system properly. And it's, it's one of those things that it's, you can't really know that unless you're, you're talking to users and, and, and really trying to get a sense of what are the real problems. It's, yeah, it's, it's, it makes me think of some conversations I've had recently, or it's like, I'll have, like there's, there's one, one of my friends who's like, oh, you should, you should do like things x, y and z to make the product. And I was like, I mean, yeah, we totally, we totally should, but those are not like big, middle, in terms of like compared to things ABC instead. And it's like, you can, it's often, you can often get into a situation in software products where, where there are a million things that you could do to improve the product. And every single one of them would be a positive improvement. And like, that's, that's not the hard part. It's not, it's not hard to know like what things are going to move the, are, are going to improve the situation. The hard part is, is knowing like, what are the things that are actually going to move the needle the most? It's like this exponential distribution of like some things are going to have like a hundred x or a thousand x improvement compared to some other things that take the equivalent amount of work. So, and in order to, to thread the needle on that, you just, you have to have so much domain knowledge about how the product works, who, who, who is using it, what they're on problems they're facing. These kind of things are just, they're not going to be in a, in a, in a training dataset. It's kind of like you could, I, I always think about these like AI tools as kind of like a, it's like a Google search on steroids. Kind of, it's like if you could find it on Google, then yeah, you can probably use an AI tool to help you with it. But, but if you, if you can't find it, just un, un, un, un, un, Google, then like, I, I don't know. There's, there's kind of a, a, a, a limit.
SPEAKER_00:  I had to take it even a step further and kind of analogize it with a common, review or argument against spatial position. People say, why do I need to remember anything? I can look it up. With Google, people say that now it's even worse with, with AI. And it's, it's like you're totally missing the point. You don't know what to look up. Yeah. Like how often in life you're solving a problem and you go, if only I look up this one fact, I would know how to solve the problem. The point is that you don't know what to look up. If you always knew when to look at what to look up, yeah, exactly. You'd be right. You don't need to remember anything. But the point is that you're, to know what to look up, you need to have that existing knowledge base to know what you're missing. Then you fill it in. You remember that. And then you can level up. You know, that's the, you don't know what to look up. That's the problem. That's why you have to memorize it. Anyway, just to.
SPEAKER_04:  That's a great point. I haven't thought about that before, but that's spot on. Yeah, it kind of, it reminds me of, yeah, this conversation I had with somebody the other day about how, how back-filling, like taking a, taking a top-down approach to kind of try to backfill whatever information you need to learn to do a task. Like, that is so much easier if you have actually learned that information in the past and you're just rusty on it now. But you have some general mental framework for these things. But if you never actually learned that stuff in the past, then you're just so confused. It's like, it's like trying to, like you had somebody a hard core machine learning paper and asking them, like, they don't say they don't even know calculus. They don't even know algebra. They're at arithmetic. Like, what map topics do you need to know to implement this paper? Like, they're going to have no idea. Exactly.
SPEAKER_01:  Unless they've- I can relate to that so much. Yeah, yeah. Right, but if you, it'll soon be remedied.
SPEAKER_04:  Yes, yes, so listen to that. In the past, that is just so much easier to, like, you can kind of look at it and be like, oh, I know, I remember, like, these kind of manipulations, statistical measures, like, I have a frame of reference. But even if I'm not, like, totally 100% up to speed on it, yeah, you know where to work.
SPEAKER_01:  So one thing I think me and Zarda both want to get onto, because I think it's one of the things maybe we disagreed with you on from reading your work and to its post-in-things, was this idea of there being an abstraction ceiling. So the people who haven't heard of this term before, I think maybe it originated with Douglas Hofstetter. I remember watching a presentation of his where he talks about this experience, where in maybe in grad school or after he became a professor, he- it's actually really interesting here and talk about it, because he described it as a very- in a very somber way. Almost as if it caused him sort of pain, where he'd reached this level of math, where he felt like he wasn't able to progress anymore to reach higher and higher levels. So can you explain why there exists a limit? I think me and Zarda basically think, like, how can it be the case that there's a limit when, in order to reach higher and higher levels in a subject, you're effectively performing the same, like, cognitive task in terms of, like, assimilating new knowledge into your long-term memory, and then sort of operating using that knowledge in order to solve problems. It feels like it could go on indefinitely.
SPEAKER_04:  Yeah, yeah, so I would actually agree that in theory, I think, yeah, I would say I think a lot of people can go on, and definitely, if you have an infinite time horizon, and like an infinite amount of work, and I- Of course, that's like, there's a caveat is, like, if somebody has, like, working memory disabilities, or, like, yeah, the things that would impede the learning process, then there might be some serious issues there. Like, some people have trouble even imagining something in their minds. I forget exactly what the technical term for that is. But, so...
SPEAKER_03:  A fantasia or something like that?
SPEAKER_04:  Oh, yeah, I think that rings the bell. Yeah, but, like, so, assuming that there are no, like, cognitive difficulties that are, like, impeding the learning process like this, yeah, I would agree that in theory, yeah, you could just go on indefinitely. But the problem is, the reason why I think that in reality, there comes to be an abstraction ceiling, it's a result of practical constraints of life, and individual differences in the speed at which somebody acquires and forgets knowledge. So, one thing to keep in mind is that different people have different working memory capacities, for example. And people with higher working memory capacity are typically going to find it easier to see the forest for the trees, given the equivalent level of prior knowledge.
SPEAKER_00:  And just to make it intuitive for people, if you think of a working memory as a whiteboard for the mind, you can just have more stuff on it at once, so you can just make sense of more things. It's simple like that.
SPEAKER_04:  Yeah, exactly, exactly, right. So, you kind of, you have a leg up on seeing patterns and things, and just things jumping out at you. And there have been some studies into the rises. I think I remember one study about working memory capacity being implicated in people's ability to generalize rules from data sets. It's kind of like, I think the study was where people were given, people were given data points as an input output, like a function, like I tell you the number, like five, or input five, output ten, and input two, output three, or something. I guess there's some sequence of numbers. And so these numbers actually followed like a kind of parabolic curve. And so a successful generalization of the rule was that like after a list of these numbers, you would be able to kind of like predict the outputs for a bunch of inputs that would, and you would kind of know that like, oh, hey, this kind of follows a curve. And whereas a non-successful generalization would be like just deviating from that curve behavior. Maybe you're predicting something more like a constant line or like an upward slope, you're not seeing the full picture. And yeah, recall working memory capacity was one of the, it had an influence on the successful generalization ability. But back to the main thing. So there are cognitive differences in people that make it easier to acquire skills faster, or make it harder to, and you end up acquiring skills slower. Same thing in athletics, in music. It's, yeah. And these same differences, they also impact forgetting rates too. There have been a number of studies finding that in addition, it's like the people in the study who are able to acquire this and who information faster, they are also slower to forget it. So it's like a double whammy.
SPEAKER_00:  How much of that do you think comes from increased coherence that's associated with higher intelligence? So the more, it's known that the more coherent of piece of information is the better you'll remember it, even with fewer repetition. So how much of it is that, and how much of it is just some correlate between high working memory and actually better long term memory, even for incoherent information?
SPEAKER_04:  Okay, yeah. So you're talking about like, I think what you're getting at is like how much does the pedagogical scaffolding factor out these cognitive differences?
SPEAKER_00:  Right. Or just assume hypothetically that we could know as a matter of fact that after two hours we got this, whatever, lower working memory people, to the same, the exact same level of comprehension and coherence as the other people after one hour. Let's just assume we could know that. Yeah. Would it be the case that they still forgot it at different rates? In other words, is there just a correlate between high working memory and high long term memory, no matter what? Or is high working memory associated with higher intelligence such that you can more quickly comprehend the same thing to a higher level and therefore it's retained for longer?
SPEAKER_04:  Yeah, that's a good question. I'm not sure about any particular studies that have actually... It's... Yeah, I don't know. I guess the answer is I don't know. I'd have to look at the literature to be here.
SPEAKER_00:  Do you have a personal sense on it? It does seem to me that some people are simply better at remembering things. Even something as simple as a name. There's almost no comprehension in a name. Somebody says, you know, my name is Robert, and somebody can remember that 30 years later and somebody can't remember five minutes later. There does seem to be, to me, like you say, as in all things. And I think the tough part is we all want to believe that because it's our brain, if we put in effort, we can do it. But we all accept that some people are taller than other people, some people can jump higher. It's all those things. And it really is the same thing cognitively. We just have a harder time, I think, and totally understandable. I've been guilty of this myself. Accepting that there really are different limitations and natural abilities for people.
SPEAKER_04:  Yeah. Yeah, I would agree with that. I think I'm a little hesitant to just say flat out like, yeah, even if people have a good one. Knowledge, then the forgetting rates would be different. But I would agree, my intuition points to the same finding. But I think in practice, again, just making the distinction between a platonic ideal educational setting where you manage to get everybody up to the same level. And some students take more practice than others. I think in practice, I don't know that everybody really reaches the same exact level. So I'm just thinking back to when I was teaching a bunch. Yes, we actually use math academy in a bunch of classrooms. I had a calculus class. And one thing that I noticed was when it came time for this particular calculus topic, that is, it's the limit comparison test for confusion. And there are convergence of sequences. And so the idea was like, you can have all the foundational material filled in. There are some students that are just naturally going to pick up on the trick that's introduced in this topic. The trick is like, you have to compare one series to another series. It's mathematically similar in a very specific way such that it has the same convergence behavior. And you would think that that's something that kind of, it's one of the things that's not a very specific, explicit background knowledge topic that goes into that. But it's something that you have a gut feeling about. And if you don't, then you can be kind of taught why that is. But it's one of those instances where some students, they just see it and some other don't. Even though they've gone through this same curriculum, they've reached baseline levels of mastery and all the topics leading up to this. So it's like, even though they've gone through the same curriculum and in theory they have the same background knowledge, there are some level of embedding connectivity in this background knowledge that the faster generalizers have just managed to imbue beyond the scope of what was expected from the curriculum. It's like you take a student who has a high generalization ability, it kind of feels like you are just reminding them of things they already know, except you can't pinpoint a time when they already learned it. Because they haven't explicitly already learned it. I guess it's kind of like these learning experiences kind of like seep through more than what you would have expected just looking at the curriculum. So I guess I think it's just, yeah, again, if you had two students with exactly the same background knowledge, not just in terms of had completed the same things in the curriculum to the same level of mastery. But like they had literally like their, if you looked at the connectivity inside of their brains and like transcribed that to see like what amount of information is this compared here. I don't know what, yeah, I don't know if they would end up like having same or different forgetting rates or what, but I think getting to that point is just something that doesn't really end up having the same or different getting rates or what, I think getting to that point is just something that doesn't really happen in real life. Yeah, because it's always going to be managed.
SPEAKER_00:  Yeah, I'm just going to say I think that's pretty intuitive because we all know interpersonally that a lot of what it means to be intelligent is that the first thing you think, or the first thing you say if you're observing someone else, is smart. It's better than what other people say. That's a lot of what it means. It's just your first thought. It's the first thing you think. You weren't going through any processing. You weren't connecting into something else. And you can also observe it in like little kids. Like I have a niece and nephew and like there's little kids around sometimes where I'm at the house hanging out with them. Like this is not to be rude to any kids, but some kids are very much brighter than other kids. They are like little kids, like two years old. They're just smarter than other kids. I don't know what that is. It's obviously a little bit depressing. If you're not on that smart side, I'm not trying to make it like that, but I think it is something we all know. Anyway, sorry James, go ahead.
SPEAKER_01:  Well, it's very related to what you were saying. Justin, you and the math academy team have put a lot of work into sort of applying all of the current best research and ideas from the literature into math academy. How many more orders of magnitude more improvement do you think we can expect? Like let's say over five years, then ten years, maybe even a century in terms of improving the rate at which people learn, either getting them to their abstraction ceiling faster, or hopefully even raising their abstraction ceiling such that they can be as good as an amazing mathematician would have been a hundred years ago.
SPEAKER_04:  Yeah, that's a great for me estimation question. Yeah, how many orders of magnitude can we hire at? Yeah, it's... I don't even know what the scale of measurement is for this thing.
SPEAKER_01:  This is something that really have a very intelligent synthesis. It's difficult to talk about.
SPEAKER_04:  Yeah, I'm trying to think how to like what does one unit of intelligence mean and what does it mean? Yeah, it's kind of weird. Yeah, let me... I do have a... it's a great question and part of the reason why it's great is because it's forcing so many good secondary questions as to the meaning of it. Yeah, but yeah, I have a lot of thoughts on that. So, yeah, definitely right. So the... just to serve with some obvious things that I know you're thinking, but it's a layout explicitly. What is like, yeah, the point of instructional scaffolding is to, yeah, make... to try to help people overcome these cognitive difficulties that would otherwise keep them from learning new information. And that applies to everybody, not just like... not just slower students or faster students. It's like the point of instruction is just to like accelerate your ability to acquire information. And you present it in a... in a way that is easy to ingest and you give a practice environment where it's just very efficient. You're knocking out repetitions on skills incrementally at the edge of your ability, building things up. And this definitely works because you can make it... you can imagine, like back just several centuries ago and then it's time like Isaac Newton and... And when calculus was like a big thing, that was cutting-edge math back then. And there's like plenty of math-y people, very serious mathematicians who never got to that point before then. And now we have high schoolers learning calculus. So this, yeah, it's definitely speed things up. And how fast, how much higher can it go? I think if I were to just kind of spitball here, well, just looking at math academy's original in school program, where we had kids who are like mathematically bright, definitely like the kind of like the top of the honors classes. Essentially they would go through this radically accelerated curriculum and they would do... They would learn college level calculus, AP calculus, BC, and eighth grade. And in high school they would learn pretty much the equivalent of an undergrad math degree. And these were not... I know that sounds crazy, but like these kids, they're not... it's not like we did some national talent search and picked the top ten math students in the nation and then put them through the math ringer to produce some tarry towels. This is very much just these kids locally selected just the right, but maybe the top five or five percent. Yeah, yeah, top five or five percent. Where are you based out of?
SPEAKER_00:  I'm in Southern California, I'm in San Bernardino, so that's just 30 minutes away.
SPEAKER_04:  Wow, yeah, yeah, that's cool. Right, yes, this is all happening at Pasadena. Nice, yeah. These students are... they're kind of their... basically, if you just took a normal high school and maybe took the top... or normal high school or middle school, whatever age range we're looking at, if you took the top five or ten percent of math students, they're probably five percent, top five percent, and put them through on the math academy system and held them accountable for doing the work and just goofing off on YouTube and stuff. Then, yeah, this is totally possible, this level of acceleration. So, I think... Yeah, I would love to see just that being a little more normalized for the... Yeah, the kids who just really are into math and are serious about learning it, you don't have to be a math prodigy, you just have to be... If you want to reach that level, yeah, there's some level of talent involved. Not a ridiculously extreme level of talent, but if you're... if you are kind of like a good, dead, mad person and you're motivated and you go on the math academy system, it's within the well within the realm of possibility to learn essentially all of high school math by middle school and then, yes, study undergrad math in high school. That's not a big ask, and I think actually that for students who are very serious, they could go even further. So this, when I say about students in middle school learning, AAP calculus, BC and then in high school doing the equivalent of an undergrad math degree, these students, it's not like they are working like four hours a day on that. No, it's just like it's a normal course. They have essentially like 40 XV of homework, so that's equivalent to like 40 minutes of fully focused work roughly, give or take, and they're doing that in class. If they are really, really focused on tasks, they can get like all their homework done in class and does that in all of our... sometimes they like the socialized a little bit more, have like 15, 20, half an hour homework, but it's like this is not... the work is intense, it's intense training, but it's not like an all day training event. And so you can imagine like take some, like if this is how far they went, just based on essentially replacing their classroom experience with a much more productive form of learning. And they got this far, you could probably, like we were talking about earlier, how much, how much deliver practice, how much taxing learning activity can somebody take before it gets into the over-training problem. You could probably go up to like more than 40 productive minutes a day, you could probably, you could definitely do a full hour a day, you could probably pull off two hours a day, beyond two hours a day.
SPEAKER_00:  And then you could do a weekend and you don't skip summer.
SPEAKER_04:  Yeah, yeah, exactly. That's big. Yeah, weekends. That's a huge... right. So you could do, if you did like two hours a day, every day, including weekends, don't skip summer. That would just... yeah. I don't have the computations worked out, but yeah, you would learn all this stuff way, way, way faster. So I think so, okay. Go ahead, yeah, sorry. Oh, so I was just gonna, okay, so my response to James, like what would be the order of magnitude of the speedup? Like how far people could... well, I guess what we're talking about here is how, what is the speedup? Whereas you were asking about the actual ceiling originally. Well, I would say the speedup, yeah, there would be like, say, an order of magnitude speedup. I think in how quickly people could reach a certain level. And I think that also would carry over for a lot of people to a higher ceiling as well. Now, I think this is where, again, I think it gets a little hairy. And that, so some people, they're gonna learn a bunch of math, and they're gonna be like, well, I just love math. I want to keep going. I want to learn even more even more. And so their ceiling is definitely being raised at how much math they're gonna learn in their life. But there's also some people who just kind of, they want to learn enough math for the thing that they're doing next, whether that is like going into engineering or maybe they're just learning up to calculus. To get into med school or something. Sometimes there's just an artificial ceiling where they'll just reach that ceiling faster and then they just choose to say, well, I don't really need to learn more math, so that's enough. So I think, but I guess probably what we're talking about in particular is the people who continue learning as much math as they can through the course of their lives. And I would say that would contribute to effectively raising the ceiling. Now, at some point, though, there comes a, I think it happens in undergrad math and once you start getting into beyond that, into grad school math, there's kind of like, you kind of run out of road and that, there's not really a whole lot of good instructional resources out there. Once you start reaching the edge of a field, like this speed up that you were getting due to your optimal practice environment due to the scaffolding involved, it starts, you enter into an arena of more friction. But like, it happened happening now in machine learning, for example. I remember back when I got into machine learning when I was in undergrad, it was like 2014 or so. It's a freshman undergrad and I was really interested in neural nets and convolutional neural nets were just, that was a fairly new thing. Well, okay, technically not new, they'd been around for some time, but they were not very, like, this whole thing was not very, not as popular back then. There were not so many like courses on this sort of stuff. Nowadays, like, you want to learn about convolutional neural nets, like, there's so many different courses out there. Like, that's like, it's almost like baby stuff compared to the current state of the art. It's like, it's so in the past, like, oh, I've been there, done that. Just learned from these resources out here. Now a lot of like, transformer is now, but yeah, so it's like, what do you start hitting that zone where there's not so much, yes, so much of a scaffolded practice environment. It just gets much, much harder to make progress. And you start, you're, so yeah, you're sealing, your effective sealing there is not going to be raised as much because you're starting to trek through the weeds.
SPEAKER_01:  Because you're basically at the frontier of what we know. Yeah, at that point you're, in order to make progress, you need to create actual new knowledge that doesn't exist out there. It's not a matter of just learning.
SPEAKER_00:  Or even if it is known, the educational resources are so bad that the only people who can pick it up are the people who have that natural sense of what's going on. Like you're saying, you're just kind of reminding them of things instead of teaching them a whole new thing. So it makes sense that some people would be better at dealing with low information environments.
SPEAKER_04:  Yeah, that's a great point, right. Because yeah, the people at the edge of the field are typically, it's slanted towards the people who had an easier time getting there, which is slanted towards the people who had advantageous individual differences. And so it's like a kind of, it's a compounding effect, right? Because you're going to have an easier time slacking through the mud of the war pedagogy if you just have those advantageous differences.
SPEAKER_00:  But it sounds to me like you're saying that if the math academy had graduate level courses that more people would have an abstraction ceiling that is in that range rather than tops out at some high level undergraduate or something else. Is that correct?
SPEAKER_04:  Yeah, yeah, I would totally agree. I think, so I would say the abstraction ceiling is kind of, it's almost like a, it's like when do you accumulate enough friction in the learning process that you just decide, screw it, I'm doing something else, I'm jumping off the shit. And so for some people, the amount of friction that accumulate, like they have a really hard time learning even algebra. And that's kind of, after algebra, they just kind of jump off the math chain. For other people, it's calculus, for other people, it's undergrad math, but it's basically, yeah, it's about when does the amount of educational friction you're experiencing kind of come up and offset your ability to acquire new knowledge. If you have a faster rate of skilled acquisition, yeah, you'll get further, but this friction will eventually eat you alive. And at the edge of the field, it's like, well, the friction is even eating the most prominent mathematicians alive. It's just that the friction comes from having to produce the new knowledge, but it's still friction. But I think a lot of people end up jumping off the train well before the time that that learning completely grinds to halt. And then the reason is because it kind of gets to a point, it's like a trade-off of, you're always wanting to optimize your time, get the most reward in life out of what you're doing. And once you get into a point where the amount of friction that you're experiencing is causing you to have to work unreasonably hard, and there are other people who find it easier and they have a competitive edge against you, then you just kind of switch to something else that you might be better at. It's like maybe, initially, you're very good at math, and you're like speeding through algebra and calculus, and then maybe in undergrad math, friction starts to really set in, and you realize that's a common story. Yeah, exactly. You're now in a pool of people who are kind of at your level in terms of acquiring this new knowledge and retaining it.
SPEAKER_00:  You used to be the smart guy and now you're just a guy.
SPEAKER_04:  Yeah, right, exactly. Yeah, the level of competition changes. So now instead of trying to lean more into like more and more math, you start looking in other directions. Like, oh, hey, I'm actually pretty good at coding too, and a lot of these people won't compete against. Not as good at coding. So then you start leading into coding a little bit more, and you kind of take a re-endering walk.
SPEAKER_01:  So it seems like friction can kind of be divided into two different things. So you can have friction that's caused by things like they're being poor resources, or schools are applying terrible learning strategies. You're missing the summer and forgetting everything. There's also the kind of friction of just your interests that align with what you would be studying if you continued learning math anymore. And it feels like if it's that kind of friction, it's kind of positive because you're just deciding of your own volition. Like, actually, this doesn't interest me anymore. I'd rather do something else, in which case that's fine. If it's the other kind of friction where it's just the resources of holding people back, I feel like that's the kind of bad abstraction scene where we've kind of failed to help people who otherwise might have wanted to go down that route.
SPEAKER_04:  Yeah, I would pretty generally agree with that. Though I think also that the level of interest in the material, I think there are ways to try to raise people's ceilings there also. I'll admit that right now, math academy is pretty, we've been just so focused on solving the training techniques problem, and it's like the kind of nuts and bolts optimizations. But one thing that we want to lean into more in the future is trying to optimize the motivational process too, and try to control people on the train.
SPEAKER_01:  I did mention to Zonda that I felt like math academy was kind of like a bring your own motivation platform. Like, as long as you have the motivation, then it's really good, but you do kind of have to provide it yourself.
SPEAKER_00:  Yeah, I don't know though, I want to mention this important thing. I think effective systems increase your motivation by a lot because you're getting results. Like, go back to your example from the beginning, the kid who had to count on his fingers. Contrast that with a kid who's already mastered his multiplication tables so that when he is learning exponents or she's learning exponents, you already know all this so it's bang easy. Of course, your motivation and your interest for learning more mathematics is going to be higher because you didn't hit a wall. Of course, I know how to do that four times four times four times four, no problem easy. That person who only by dent to the fact that they had mastered the multiplication tables earlier on is now going to be more interested in further learning of mathematics. Not because of inherent difference, but because of a difference in effectiveness of what they've done before. And math academy is so effective that it does seem to me that you would get a boost in motivation and interest simply because you're actually going to be able to do that. Simply because you're actually making progress. Oh, by the way, I'm sorry. I just want to mention this. I know we've got for about an hour and a half. If you have to go, just please let us know. We don't want to go an hour. We've already got an hour and a half. I just want to be a specialist.
SPEAKER_04:  I actually don't have any pressing commitment. So I'm good as long as I want to go for. Yeah. I could talk about this stuff for like hours even.
SPEAKER_00:  Yes. There's so many things I want to ask you.
SPEAKER_04:  But I just want to say I totally agree, Xander, with the idea that like I think that generally what you're getting at is like nothing succeeds like success. That's right. Yes. Able to succeed and make progress is the number one biggest motivator when you feel like you are growing. So I would agree that in that sense, yeah, Matt, Matt Academy does. I guess technically does have that as a motivational technique. But there's still like, there's a lot of other stuff that we could be doing. Like streaks, for instance, and like just like notifications of like, hey, if you set a practice schedule for yourself and you're meeting to do some math every day. And it comes tonight and it's 8 p.m. you haven't done any math, like just send you a little email or a ping. Right. Like, hey, just do one lesson. Do one lesson tonight to keep the streak going. Something like that. Do one and go does a lot of stuff pretty well. But I would say one thing that makes it a motivational problem a little harder for us as compared to other resources is that like we are not pulling punches in terms of all of that. And how much material we cover and how rigorously we assess you on it and how if you don't do a good job on it, we tell you. And we make you do it again, like later on. We make you do another try on it. And it definitely takes like to succeed in a practice environment like that. It takes more motivation than to go through like some water down, shallow, perculum. That just lets you miss some problems and then just move on to the next thing, what's you like fail for. It doesn't like have time to quizzes and stuff like that.
SPEAKER_00:  So the FAQ page is extremely modest on that point, at least from my recollection. There's a question something like, is this more comprehensive than the average course and it says something like it can be. And I just thought, what a modest way to phrase it.
SPEAKER_04:  Yeah. So the reason why we do a lot of that kind of modest phrasing, because it's like, well, okay, on one hand, everybody likes somebody who is kind of humble. Who does a great job and stuff, and just says, yeah, I think I do a pretty massive job. That's just makes it more friendly, more approachable. But also, there are really some really intense college courses at places that go into very deep coverage. And so it's like we don't necessarily want to declare that literally every other resource sucks and that we are the best. Because there are some other classes that actually do a good job. But yeah, it's pretty, by far, the default mode in school is very lacking comprehensiveness. And I would agree that compared to your typical school, compared to almost every school class, we kind of blow them out of the order in terms of comprehensiveness. And maybe there are some that kind of like are more on par. I still have a feeling that it would be hard to find more comprehensive courses than what we have. We do textbook comparisons. We try to get every, like, a superset of all the key information, all the key textbooks. But yeah.
SPEAKER_00:  It's really incredible. You don't have to agree with this, obviously, because of your immense modesty. But it's really an incredible thing. I mean, just look at the knowledge graphs for any one of the courses. And you will see the depth and the breadth of all this and how much work. I mean, it's a sight because it has to be done manually. Everything in math academy is interlinked. It all has dependencies such that it knows if you get this wrong, maybe you're weak on this point, weak on that point. Or if you're strong on these, now we can introduce you to this. And so on, it's all to do with the layering and the spatial repetition algorithm and all the rest of it. So it's all linked. It's so in depth. It's so fine-grained. And that has to do with even more mechanisms of learning, like we were talking about, working memory, sub-glot, labeling, knowledge points, you guys have. I would love for you to get into more of that if we can. Also, I think James may have had a question. But I just wanted to say it's amazing. If you're listening to this and you haven't checked it out, like seriously, just check it out. I'm not trying to sell you on it. You don't have to get it. Just check it out for your own interests. If you care at all about learning or these ideas or what's possible in this space, just go check it out so you can see what some people have done. This team of people that have clearly put so much into this that it's hard to even comprehend. I mean, it's truly actually amazing.
None:  Appreciate it.
SPEAKER_04:  All the kind words there. We've put so much work into this thing over, I mean, Jason and Sandy, it's been over a decade since the initial conception of this. Alex, I think, has been working on this for seven years. I've been working on it for six or no, five and a half maybe. But it's like just around the clock. You just took a bunch of nerds and put them in a room for like 80 hours a week. They're just going to town, coding stuff up in their little nerd world, space representation and math and stuff. And then five years later, you check in on seven or ten years later and you check in and then you see what they come up with. Yeah, it's a lot. It's a lot. It's incredible. One thing I would want to say, just on the tie-in, what you said just now along with the, just the whole comprehensiveness is one thing that people sometimes try to point out is like, well, hey, you guys don't have a proof-based linear algebra course. How is that comparable to Axler's linear algebra done right? How can you guys claim that your comprehensiveness, if your linear algebra course doesn't have all these proofs in it? And the thing is, we are building this curriculum up from scratch, from bottom up. And our current linear algebra course is really something that would be a prerequisite to something like Axler's linear algebra done right, an abstract or proof-based advanced linear algebra course. Oftentimes, universities sometimes throw students into Axler's book without previous knowledge and linear algebra, and it is just a major struggle for a lot of students because they don't have the prerequisite knowledge in terms of how these procedures work under the hood. They may not even be comfortable with how to compute an eigenvector or an eigenvalue or diagonalize the matrix, and now they're being asked to do proofs that require intuition about these processes. I got through the first chapter of that. What was that?
SPEAKER_01:  I got through maybe the first chapter of Axler's book, and then it was like, yeah, it comes up on you pretty quick.
SPEAKER_04:  Yeah, you can kind of get to thinking that, oh, you just can't handle the material. In fact, it's like, no, there's just this huge chunk of prerequisite material that you're missing. And if you had that chunk in, you could totally get through your axler's book. So what I'm getting at is like, it is 100% on our roadmap to also have a second course in linear algebra, which is really what Axler's book should be called a second, linear algebra done a second time. That would be a more suitable title for it. But we very much had a roadmap to have that course. All of you, pure math, or not you guys, but like all of these pure math focused people who get very like high and mighty on their ivory tower of like, oh, we, we do all these proof things. You guys don't do as much proof things. Just know that like, math academy is coming for you. We're not here yet. We're still building out what we just released our initial methods of proof course, which covers all the basic proof techniques that unfortunately so many people who may join math don't learn beforehand and then are just kind of thrown into the deep end and becomes a source of struggle. But we've started rolling out our first proof course. There's going to be more proofs courses, more advanced material. So the, yeah, anyone who claims that math academy is not going to reach the full comprehensiveness of like a true undergrad math degree. Like at this moment in time, what you say is true, but this is going to change in the future. Yeah, we're going to, we're going to cover everything better than anyone's. I love that. That's awesome.
SPEAKER_00:  I want to ask a few quick, quick and quick questions, but I just want to make sure James, you have a question.
SPEAKER_01:  I do. Can I sneak in before you? Yeah, yeah, go ahead, please. I've been talking too much. I'm curious to know about the roadmap and specifically there was this cryptic and quite exciting tweet from Jason on Twitter. He said, one of the most exciting yet stress inducing aspects of running an early stage startup is that occasionally opportunities will present themselves that have so much potential that you have no choice but to blow up your short term roadmap ignore any number of raging fires and take a shot. Now you don't have to answer specifically if it's a secret source or anything like that, but I'm curious to know what's on the roadmap and if you could hint at what Jason's may be alluding to that.
SPEAKER_04:  Yeah, yeah, I can talk about that. So the idea is basically we had this whole, well, we have so many things to do and we were going to do a lot of the things that we were meeting to do. And then in August, we got contacted by a couple schools who are interested in using the system. I can't say the names of the schools, but one of them was like one of the, I think Jason mentioned on his podcast, the way he described it as one of the top five leading institutions in the US and in the world in general. And so the idea was that like, okay, well, when that happens, when these kind of people are kind of sourcing, like, trying to, trying to, trying to do business with us like we, everything else like whatever it was that we were supposed to be doing has to be put on hold and we need to build out like some stuff that they need. So the general idea was like, yeah, we had a push towards building, building some functionality to help integrate the platform better with schools. And there were a lot of things that we were wanting to roll out more for individual learners, who is our by far main market right now. But just want to, yeah, when an opportunity like this presents themselves, just going to have to let a bunch of other buyers for while you try to capitalize on. So that's the subject.
SPEAKER_01:  Yeah, that makes sense. Is it difficult to balance sort of the competing priorities between the lifelong learner or professional sort of users and the schools? Or is it kind of overlap?
SPEAKER_04:  It's, it is actually kind of difficult. I mean, they're not at odds with each other in any way, but it is kind of orthog. There is some other overlap, but I guess so some ways in which it's kind of orthog and all is like, if you are a teacher and you're trying to, you're having your kids in class use math academy. Well, you also need to make sure that everybody is staying on task and just being aligned with the process. So you need some kind of dashboard that you can look at that has some aggregate stats on each student. Also like how long ago since they last solved the problem or last active. So you can tell like, hey, Johnny, it's been, it's been over 10 minutes since Johnny like solved the math problem. He might need help on something. Or like, that's the best scenario or scenario is like, oh, Johnny is just over on YouTube messing around. So you got to have a way to keep your hands on everybody. And of course, that's something as an individual learner, you just, you don't need that. But there are other aspects that come in useful in terms of, I think, the communication aspect. You can kind of, this kind of goes into a lot of the things that a teacher needs to know about their students. Line up pretty well with the things that a parent might need to know about their kid on the system. A lot of our learners are, their individual learners, but they're a kid whose parents sign the law. And the parent is essentially playing the role of a teacher in a one student class. And the parent, the parent doesn't have to teach the subject knowledge, but they have to, or they have to be like the coach essentially for the kid. Make sure that they are aligned with the whole learning process. And if things are going off the rails, then they need to know what's going on, how to fix it. Yeah. And then of course, even for individual learners too, it's like, it's, we also just need to do a good job of communicating just how much progress you're making. Sometimes it's easy to get lost in the grind of like you're solving math problems. And like you know, you probably know like deep down that you're getting better, but it's like nothing is ever feeling easier for you because you're always just pushing at the edge of your ability. And you're just kind of like, it's easy to get a little demoralized with that sometimes. And so it's, it really helps sometimes to just have like very concrete visualizations and metrics that are showing just your improvement over time. And even something like, like something, an idea that I always thought would be cool is like, it's some, some, some milestone celebration like, oh, it's, you've done like 1000 XP on the system. Look at the types of problems that you were solving originally when you first started out that were kind of challenging for you. Like how did these feel now? Well, pretty easy now. I would always do that with actually in the math academy school program. I taught this really intense computer science program and kids would come in not even knowing how to code. And, and, and we would just power through your basic introductory coding. Some, some basic algorithm stuff building your own linear regression model logistic regression model, decision tree, neural net, back propagation, wow, narrow evolution, reimplementing papers in the, in the. Blondy 24 AI program back from the 90s where this guy trained a neural net to play like tic tach toe and then, and then checkers and stuff. So we would, we would. The kids were like, they were always, always working tackling these hard problems. And sometimes it would just get to the point where there's this like, oh my gosh, can't anything be easy. Why is everything so hard? And, and, and the way I would, I would often just kind of like help motivate them again is by reminding them like, hey, look in hindsight things things only get easy in hindsight. And, and look at the type of stuff that you guys were doing at the beginning of the year, like this problem that you struggled with. And, and we're, it took you like the whole class to do how long would it take you now. Oh, and it would take you like five minutes to code this up. Wow, that's, that's a big difference. And so just seeing that kind of, these kind of really tangible feelings of improvement. So, yeah, I would say just the general link between schools, parents, individual learners is communication of progress and motivational, but also like keeping things from, from going off rails.
SPEAKER_00:  Right. I imagine that's, it's a bigger problem for people who are using like maybe adult or using math academies there. Primary or maybe even only source of engagement with mathematics right now. So you, you don't never encounter a problem that you've already learned how to solve. Like you said, you never are using your knowledge even beyond the platform yet. So I can imagine you, you would feel a bit like man, I'm not. It's always difficult, which is if you're learning effectively, like we're talking about desirable difficulty and all the rest, it is difficult right on that edge. So that's a, that's a tough problem for sure. I did want to ask a few more curriculum questions. These would just be quick ones. You mentioned that the, the kids in the early program were able to go through the undergraduate stuff in high school. So how much of the undergraduate stuff is done and when do you think it would be a full math undergraduate degree that's available on the, on the site?
SPEAKER_04:  Yeah, a great question. So right, right now in terms of undergrad math, let me just enumerate what we have. We have calculus one, calculus two, multivariable calculus, linear algebra, methods of proof. We have a math from machine learning course that, that pulls in all the necessary linear algebra and multivariable calculus and also has quite a bit of probability of statistics that we've built out for that course. Now, probability of statistics, we don't have a full course out yet, but that is one of the upcoming courses that we're working on currently.
SPEAKER_00:  I think it says it's going to be released this month. Is that my memory that way?
SPEAKER_04:  Yeah, I think that was a little, we spoke a little too soon on that. I know, I know that's like, I think that's the next course up. But yeah, you'd have to ask Alex, our curriculum director. He's Ninja underscore of math on Twitter. He's just like me. He's very, very responsive and friendly. I'm sure he's happy to give an answer to that. But yeah, I'm not sure exactly when this is coming up. It's funny, we originally had some estimated completion dates that we had not really blown up on X Twitter at that point. And so we were not getting really any flack for being a little late at those. So that now, like once we kind of blew up a little bit, now we have like, oh, it looks like we've written a couple checks with our mouse on the timeline that we're struggling to catch.
SPEAKER_00:  Well, at least there's enough people using it to get upset. That's a good problem.
SPEAKER_04:  Yeah, that's true. Yeah, the more people we have using it asking for these courses, the more pressures on it. To give them out the door fast. That was one of the, like Alex posted recently about hiring content writers for Math Academy. Because he's, right now he's kind of a bottleneck in terms of like so much, all the content, because it kind of has to go through him so he can just give the stamp. I'm like, yes, this is good, this is quality. Because we don't want to like over delegate and put out content that is hard to learn from. Because that's just not us. We can't afford to do this. So, yeah, but we're trying to speed up the development process. But in terms of like, so other courses that we were also working on discrete math as a course, that's going to be a good process. That's going to be, I think, a really interesting one when it comes out. Because it's going to leverage tons of proofs. Methods of proof is actually going to be a prerequisite to discrete math. So, discrete math is going to go very serious on proofs. But it's also going to have like graph algorithms in there too. And it's going to kind of be our first kind of foray into some of the more computer sciencey concepts. So, that's going to be a fun one. But what else? Well, yeah, abstract algebra. So, we have some abstract algebra content that we wrote for the school program. And now we haven't finished the full course in abstract algebra. And the content that we've written is, I mean, it's not bad, but it could be improved. And when we actually launch an abstract algebra course, we'll want to improve the existing abstract algebra content. Right now, how that functions in the system is kind of like, we also have some differential equations content that's in a similar state. It's like, it's not bad, but is it enough to, like, when we release our differential equations course, we're going to have to re-man the contents a little bit. But right now, yeah, we have a bunch of university level math, like differential equations and abstract algebra. That's kind of floating around in the system. It's not in a released course, but there are topics that are technically live and kids from our school program were using them. And if you reach the end of the road in math academy, you do like all the, say you're in the foundation sequence, and you go up through your foundation sequence, foundations one, two, three, you do your math for machine learning or whatever high level course. And you're kind of like at the end of the road of your course sequence, then you're going to start getting a hodgepodge of just, if you stay in that course, well, there's nothing to promote you to. So unless you switch to some other course, if you just stay in this course that you've completed, it's the top of your sequence. And you're going to get this hodgepodge of all these topics that are technically live in our system, but they're now part of some officially released course.
SPEAKER_00:  That happens by default, it just blows you all around. Yeah, yeah.
SPEAKER_04:  Now these are topics that are, again, they're like, they're in a good enough state that our high schoolers in our school program were able to learn from them pretty well. But they, we will definitely want to revamp them before it's like in a publicly released course. But it's kind of like, yeah, it's because we kind of, we want to try our best to prevent people from running out of things to do. So it's like, well, there's some stuff that's in a workable state. So yeah, let's give them that once they complete all the topics in the system. We actually, just, just the other day, my, my model broke an edge case that I always wondered when it was going to come up. Some, some guy actually completed all the topics in the system. Like we just, we just, including hodgepodge, like after I down through it, the differential equations, like we just, we, he keep the game in one. It's, it's not completed now. Yeah. Exactly. Until, until we build more courses, more levels into this thing. But yeah, you know, just getting some review and all the stuff that he's, he's done so far. But eventually, so it's a later down the road in addition to those, those courses, like, like discrete math, probability statistics abstract algebra, differential equations. So we need a real analysis course. We'll need a complex analysis, probably like a number theory or graph theory. We want to cover like an entire undergrad half degree. That is, that is one of our key goals. So we're still always off from that. We're still in quite a few courses to get in place. I think, I think once we have real analysis and abstract algebra in place, then, then that'll be like really, it'll, it'll feel like a, like a true, full undergrad math degree. Because those are like the two core courses that, that are often seen as the hardest courses in the, in the, in the math degree. Oh, I guess the politics too.
SPEAKER_00:  I think an important point for people who are maybe using that as a criteria, if, if that's what they're trying to get to. You know, it's not just that, yeah, maybe they don't have it right now and they, they will eventually. So that's one point. But also, it gives you a good base. So if you want to go learn that stuff on your own, you're in a much better spot. I would imagine.
SPEAKER_04:  Yeah, 100%. I, I actually, when we released our methods of proof course, you know, I, I, Alex is the one who's like, it's very heading. And this stuff needs a little bit of building the course. So I hadn't really seen a whole lot of it. I was focused on my model stuff, but I took a look at it and I was just floored at how much I wish I had this thing before I went to college. And I would, it actually came right after like the week after I had this conversation with, with this guy who, who attended University of Chicago. And he started out like wanting to do math there, but he realized pretty quickly that, hey, just getting a, a five on eight p calculus BC was nowhere near preparation for these math courses with all these like crazy classmates who had been studying a lot of math a lot more seriously. And many of them already knew a bunch of multi variable cap when the algebra tons of them had already had a lot of experience proof writing proof writing is like one of those things that, that it's kind of unfortunate, but a lot of math programs, especially like really at really prestigious schools. And just kind of, they just kind of assume that the students should be able to pick it up on the fly, which unless you are like some mathematical genius that typically does not work out very well. And you might get through it just, but it's going to be like really a rough experience. And I think if every, if every high schooler who's an aspiring math major, and they took their, their eight p calculus course, if they just took this methods of proof course that we have out that it covers all the zoo of different proof types, proof by just direct proof, proof by contradiction, proof by contra positive proof in various scenarios, proofs of the visibility, proofs with modular congruence, proofs, level of all these different sorts of cases in which you might see proofs and different like phrasings of things. So we go into like quantifiers, like for all for any, and like all the logical manipulations of these things, the truth, the other way, you just, you come in, if you do these things and you, we also practice applying them to, to like early, early topics and like real analysis or abstract algebra, like in our methods proof course, we have absolute delta proofs, get tons of experience with that. And abstract algebra, we got tons of properties of the additive and multiplicative groups of integers modulo n. And if you come in with that background knowledge, you're going to be so well set for anything that your math program might throw at you. And even if they don't do a good job of filling in your background knowledge on proof writing or on how quantifiers work or, or they just, or maybe your teacher actually had a family member had had the worst course on real analysis that I've, that I've ever heard about last spring. And I had to, I was, I was tutoring them through it and just the, there's no notes for the class, no textbook. The teacher just goes up and like writes down some epsilon delta proof. Doesn't even talk about how to work backwards and find the right like epsilon for this proof. Like typically what you do is like you, you work it out on scratch paper to kind of work backwards. You figure out the right value, the right expression for your, for your epsilon that you have to plug into the proof to make it work out. This kind of just like pulls it out of a hat. And he's just like, well, you know, if we choose epsilon it goes well, that should work. And then like, and the, the, the class just is like, wait, where, where'd that come from? He's just like intuition. And it's just ridiculous. But so the idea is like, if you come in with these foundations filled in, you kind of guard yourself against situations like that happening. And you get the crappiest teacher for real analysis who doesn't have a text, who doesn't use notes, doesn't even explain how to do stuff. So everything is intuition. You can still get through the course and it's not going to derail you from the math major. And I think, I would, I would say that if, if every high schooler was aspiring to be a math major in college, just took this course. But this is proof. I think it would cut them. The math major drop rate by, by more than half. I've seen so many instances where it's just like people, people have no, no, no problem with the level of interest in the material. They're like really, they're really interested in it. They're really interested in math, but they just get exposed to some of the worst teaching practices in college. And it just, it creates so much struggle. It totally sours their experience. And they just, they just think like that they can't pack it when in reality they're, they're just missing prerequisite knowledge.
SPEAKER_00:  Yeah. And that's, I always think that a truly bad explanation blames you for its badness. It's like, it makes you feel like it's always your fault for not as many. And that's a big problem. It's the default assumption. It's my fault that I can't have said it. I'm somehow stupid or inept. And that's why I'm not that this course is bad. That's extreme. And I'm just thinking, because I never went to school. I don't mean university. I never went to any school. And I'm just thinking, if I had had this when I was nine years old and I could have started on math academy, with my personality type especially, it would have totally changed things for me. I would have been immensely grateful for it. And it's amazing that kids have access to it now. And honestly, I hope that, that it becomes the norm for kids to do this. Not as a force thing, but because it actually enables you to be capable with this, not be afraid of numbers and actually do what you want to do. Even if that's not extending beyond what's available now. So I'm just super grateful that it exists.
SPEAKER_04:  But yeah, this product was born out of the same sentiment of like, just me and Jason, I think to some extent, Alex, we've all had to, we've all been in the situation where we've wanted to learn a lot of advanced math. And we just had to teach it ourselves because the school systems that we were in were not, were not supportive of that. And yeah, when you just, when you try to do something like that, you come face to face with a realization of like, that everything is just so inefficient. And yeah, you want to be elevating your skill set, but you're constantly experiencing friction that's trying to drag you down. And yeah, it's a very emotional experience. And so yeah, this product is like kind of built out of our own, like what we wish we could have had growing up that would have changed our lives. This is, this is exactly it. I'm really interested to know, like, so you said that you didn't go to school at all. Like, what was your, like, what was your math back or like how, how did you like learn, like, what was the, the, what, what resources were you using to learn math or were you using any or like how, because I'm just curious, I don't really know a whole lot of your back.
SPEAKER_00:  Yeah, yeah. Yeah, it's the math part in particular. I never learned much math. I mean, even now my mathematics is very weak. I naturally, I was inclined to basic things, arithmetic, stuff like that. So I picked that up. Fine, I was a little kid, but I never studied anything beyond that, even like algebra. I was never, I mean, I did some kind of academy stuff of my own religion. That was, that basically categorized all my learning, maybe after the age of 10. It was all directed by just what my interests were, because I had started learning how to code when I was nine or ten years old. Yeah, I had started working with coding based stuff like actual jobs when I was 11 or 12, something like that. So I just, a lot of it was born out of my own interests. I really wish now that I had learned more mathematics. I mean, that's a true wish of mine. It has been a long time. The problem is when you're an adult and you have a job and responsibilities, there's huge opportunity, cost for all this stuff. And also, there's no great resource. To be honest, like, to shore up my math, just to give you an example, and, well, I hope this doesn't put you in a tough spot to say something negative about another system. But to shore up my basic mathematics knowledge, I have tried a beast academy. Like, I've actually done that for people that don't know. The beast academy is an extremely comprehensive first through fifth grade mathematics program. And there's physical books if you want them, but there's also a great web app for it. But I gotta tell you, even with like serious effort with me as an adult, and you know, I'm not, I'm not mentally handicapped severely at least. So I kind of want to think I'm relatively, you know, a reasonable example of what it would be like as an adult to learn fifth grade math, fifth grade. And it's like, even that, I felt like I really am not making much progress here. I don't know if it was just me or what, because I heard such great things about beast academy and art of problem solving in general. And as I was doing, I'm feeling like this is taking me a lot of time for stuff that really should be very simple, or at least to me seems simple. And just to go through all the problems, it was taking me ages, even though it wasn't introducing a lot of new concepts, at least that's how I felt. So anyway, I've tried other things like that. I've tried Khan Academy, as I said. And yeah, I'm severely lacking my math knowledge. So that's kind of what I am with that. But the rest of it was self-directed.
SPEAKER_04:  Yeah, I think I can help you out with that. I'm sure there's a lot of people who are in the same situation. Yeah, one key difference, I think, between us and a lot of other math resources that are often considered, like serious math, is that a lot of other math resources. So you seem to focus on this kind of creative problem solving. Like trying to, how you're saying, you do a problem, but it feels like it takes forever, and what did you learn out of it. It's like, it kind of reminds me of the whole issues with discovery, learning, for novices, in that it's just kind of inefficient to be in a state where you're having to kind of produce your own solution types and attempts and stuff. It's like how we talked about how things get muddy once you're at the edge of a field and you're waiting through this tarpit of friction because you are having to produce your own knowledge and you don't know that you don't have the guidance to navigate it efficiently. That happens with a lot of these resources that are trying to have people focus on math problems that involve some creative insights that they're not actually being trained to learn how to train to apply. It's just so much more efficient to just guide the learner in exactly how do you do this and have a lot of her do it and then just pull together skills with guidance as you go. The huge problem is you're saying that it takes forever and in order for this whole deliberate practice cycle to work, action, feedback, adjustment, getting better incrementally, you have to be going through a massive volume of repetitions for personal and harder things. If you are just doing, say you class spends an hour just trying to explore some really hard math problem. Say it by the end of the class that they solve it, at the end of the day that's one repetition. You go to the gym for an hour, drop down and do one push-up. They call that a workout. What even is that? That's not a workout.
SPEAKER_00:  Yeah. With the Beats Academy, in particular, as an adult learner, one of my primary concerns, just to minimize the opportunity, because here's efficiency. I'm trying to do this as quickly as possible because I don't have all day. I'm not going to school. I'm trying to make real progress in as short of time as possible. Maybe if you're a kid, Beats Academy would be great. I don't know. I have no doubt that it's comprehensive, but it's just to me, it's not what I was after. It seems to me that math academy is much more interested in that true efficiency, which is you're actually learning things. You're constantly right at the edge. You're right at the edge of your knowledge. You know you are because of that graph. There are so many parts of it that are so thoroughly thought out. I'm sorry for... I'm an abusive person in general, but I'm sorry for being an extra abusive year. So anyway, that's a very amazing thing.
SPEAKER_03:  Really?
SPEAKER_04:  You love the ground. Everybody loves the ground. It's cool. I remember, you know, there was a time initially that we didn't actually have any sort of graph visualization. This was before we, like, anyone was actually using the product publicly. But I actually... It was back when I was just starting to work with Jason on the automated task selection algorithm. I remember that I was just as a toy project. It just popped into my idea one day of, you know, it'd be really cool to visualize all these topics in some way. And then so I just pulled up... It was one of those online code emulators, JavaScript emulators, something. And I just dumped in some connectivity for our calculus course and then imported graph this library for visualization and put it on the screen. And it was kind of incredible how beautiful. I showed Jason, he's like, whoa, we need this in the system right now. And yes, and you were saying how Jason was just spending the rest of the day, like looking at it. Like, just... Jason, come on, like, look at the graph. It's just kind of mesmerizing. Yeah, and so we... Yeah, we... We want to lean into that more, definitely. It's something that we've been meaning to do for a while is just have the graph be front and center in all the visualizations, everything that's going on. Like, ideally, we want to get to a point where you complete a task and after the task, you see, like, the effect that it's had on your graph. You actually see the new topic... That's fantastic. And the space for petitions trickling down into the encompassed topics that are being exercised. That'd be awesome. And then maybe, like, on your, like, you get some, like, weekly reports or something, and it shows a time-lapse of your graph filling up. Like, what did the task do this week? This is all you... That'd be great. Your knowledge and the graph filling up. Yeah, exactly. Yeah, there's so much we could do for it.
SPEAKER_01:  It's exactly the same, like, feeling you get when you look at, like, a skill tree in a role-playing game. And you can see that. And you're gaining skills. And you can also... Like, one of the things that I was going to send a message about is that when you have a lesson, it would be really useful to know at a glance, like, what is this going to unlock for me, later down the line. So it kind of gets you excited for, okay, if I complete this, this is unblocking, like, a roadblock for me to get to, say, calculus or something that I'm really excited to get to. And it motivates it a lot more. And if you had the graph visualization front and center, it would serve that purpose. It would be really useful.
SPEAKER_04:  Yeah, 100%. That's actually... You took the words right out of my mouth. That's exactly what I was going to say. It's also on our week. We want to make it so, like, any time you get a lesson that is in, like, some pre-record, some lower-level course because it's, like, a missing foundation for your current course. We want to tell you, like, okay, why are you doing this lesson? Because you need it in order to unlock, like, blah, blah, blah lessons supporting the current course. So just, like, yeah, like, closing the loop of thought, like, how is this actually relevant to what you're trying to do, exactly? It's kind of embarrassing that we don't have all this stuff in there. Jason says that all the time, like, it's the kind of vision that we have for this system. It's like, we're glad that the system is being, like, useful and valuable to people and it's current state, but it's still nowhere near what we are imagining for it. It's really good to hear that from you.
SPEAKER_01:  Well, that's why you made it. Yeah. It's really nice to hear that you have such, like, high ambitions for it and that it's obviously, like, such an important problem to you. And it's really, like, inspiring to see. But, to be honest, like, using the system, I feel like you've gone about it the right way because you've been focusing on the content, the parts of the system that actually matter the most. Like, you're talking about the high leverage points. The content is the high leverage point. You know, having the comprehensive coverage of all of the things people want to know and having a really good space repetition system that allows you to skip a lot of stuff. Like, that was one of the things that was really important to me. You know, not spending wasting a bunch of time learning things already new. And there's an, and all of the other things you put into the system. That's more important to me than having, like, a really polished UI or some nice gamification features where it does, like, an explosion of confetti when I finish a lesson or something. You know, that's all stuff you can add later down the line. And it's nice to have, but it's not the core experience. And the core experience is, like, obviously, as, for me, as a user, it's what you've been focusing on, which is excellent.
SPEAKER_04:  Yeah, exactly. But, yeah, focused on building it, some core functionality. And even, even then, we have so much core functionality that's not even in there yet. It's, it's gonna, yeah, you know, we, we have so, so many big plans for this, for this system. So, yeah, you should indefinitely into the future. Keep getting better and better. But I'm so glad that you're on a magnitude left.
SPEAKER_00:  Yeah, right. Exactly.
SPEAKER_04:  But I'm so happy that, that, that, that, that you guys appreciate it in its current state, even if it's a little rough around the edges. But I'm, I'm glad it, like, is solving some, some key problems.
SPEAKER_00:  Yeah. Yeah. I, the world is a better place for its existence. So I'm glad you guys didn't keep it closed.
SPEAKER_01:  All right. Should we do one more question? Otherwise, we'll keep you here forever. Justin, you might never leave. Yeah. Um, no rush, no rush. I guess, as a final question, maybe you could give, like, your most important takeaway from the experience of helping to build math academy. And maybe looking back, what would you tell your former self, you know, if you'd give them advice? Um, what, what would you say?
SPEAKER_04:  Yeah. That's, that's a really good question. Okay. Yeah. Me and takeaway. I think, um,
None:  Okay. There's a couple of things that come to mind.
SPEAKER_04:  The first thing that comes to mind is, uh, reminds me of, uh, Paul Graham's essay, like, do things that don't scale. I think, um, I think one of the reasons that math academy is being able to, to solve the problems that we're, that we set up to solve to, to provide such efficient instruction is because we didn't try to, our first step in solving this problem was not to, uh, try to come up with some software solution. The first step was actually just teaching classes in person, um, and trying to just, like, get a feel, get a gut feel for what, what, what does it mean to do effective learning? What does that feel like? What are the techniques that need to be used? How do these things work? Do they work? Let's see. Let's use them in the classroom. Let's see, like, how this, how this goes. And even, even though it's like, it's not, I don't think it's possible as a, as a human teacher to, to fully leverage all of these techniques. It's just too much work. But you can kind of get a sense of, of how they work, uh, and, and get a feel for how things go, how the, how the details need to work. And when you, when you start off just solving a problem manually, by hand, you're just, you get yourself in a really good position to automate it in a way that actually solves the problem. It's too easy to, to, to build software in a way that you think will solve a problem. And then you, you invest all this work into, into this solution. And it turns out that your solution is actually defective in some way. It's, it's just off base. It's missing some core pieces, some puzzle pieces of the problem. And I think, yeah, so I, I, I think doing things manually, doing things that don't scale, just trying to solve the problem at the lowest level of scale and then ramp up the, the scale afterwards is, is the way to go. That's probably the one key takeaway. But the, the other one, which is kind of related to this do things that don't scale is, when you are, when you are doing things that, that don't scale, you're, you're in a situation where you're kind of building things up from first principles. Like you're, you're not, there's a mistake that a lot of people make that is, that is like they, they try to do something big. But it's, the thing that they're doing is so big that the only way that they can try to do it is by trying to copy other people who have tried to do big things. And, and so it, it, it ends up being like you're just pattern matching to the, the things that, that bigger companies do or things that famous people do or whatever. You're just pattern matching to some cases that you, you think are, are somewhat successful. But when you, when you go down and do things that don't scale and you, you were, you were working at the lowest level, you get to see the real levers that are being pulled. You get to see that a lot of things that are common knowledge to people who are pattern matching are not, are actually not true at all. And that is, in education in particular, that is one of the, one of, I'd say education is a setting where, where this is, is most true, but a lot of things that are common knowledge are actually not true at all. And, you know, like you can read all about these like neural myths and like the myth of, of learning styles. And, and, and there's so many things that, that people think move the needle on learning, but they're not actually measuring these things that are first principles. But the first principle is level, trying to like manually improve a student's learning and holding themselves accountable for it. They're just kind of pattern matching to something that somebody who's kind of famous said, or something that some big company does. And, and you, you, you just, you need to kind of figure the x out for yourself, figure out what are the key components of the problem. Yeah, I think the only way to do that is to start out at a level where it's actually feasible to work for first principles.
SPEAKER_00:  Very good answer. I know that was meant to be the last question, but I have to ask myself the question. I, I think other people, I think other people will be interested in this also, but it's certainly a savage question. If somebody wants to learn all the way up through just enough to like be capable at undergraduate level mathematics, is there something else that would need to be supplemented or would be helpful to supplement math academy with, or you think just do math academy and you're good to go?
SPEAKER_04:  Honestly, I think just do math academy and you're good to go. Just get to level of methods of proof. I think, so you definitely, if you want to, if you want to use math academy as your primary resource to get you up to a place where you can, you can join an undergrad math degree and just have, have no issues getting through the courses. What you would need to know on math academy is you definitely need to know all the way up through calculus. And I would say you, you also need to take methods of proof. Some people manage to stick the landing from learning AT calculus, BC and high school, and they come into a math program. They manage to keep up, but there's a lot of friction. And I'd say only half, maybe less than half of people actually manage to stick the landing. Most people who don't come in with background knowledge just have lots of struggle. So the way to prevent that, take the methods of proof course, and ideally even take the linear algebra and multivariable calculus courses too, just to be really prepared. But I would say, at a minimum, where I would draw the line is you need to take up through calculus and our methods of proof course. And then you're about ready.
SPEAKER_00:  There's no better answer than that. Just keep grinding XP. That's exactly what you've got to do. And on XP, by the way, that reminds me, think of how much we didn't get to. We could do another eight hours on this. I promise you, we'll close it out here just in the interest of time and people's busy schedules. But we didn't get to so much. And one is the XP. Can you answer this? Maybe you don't want to. I want to do the XP if I do everything in the system like that one guy did. How many XP did he have? Can you answer that?
SPEAKER_04:  I can give a ballpark. So I think, I can't remember where that guy placed in the system. He did not start at fourth grade math. I'll tell you that. He placed higher. He had some background math knowledge. So he did not complete every single topic explicitly. He placed out of quite a bit. But if you were starting at fourth grade, you did all the work in the system. I posted something recently with baseline XP measurements for all the courses. I think it came up to something like 30,000 or 40,000 XP total. As a baseline. That's for like a baseline serious student who is a good student, but it's not perfect. And the system adapts where you get more frequent review if it sees that you're struggling on some things. Every time you miss a quiz question, you get a follow-up review. The amount of XP kind of depends on how well a student is doing on the system. But just for baseline serious student, it's imperfect. I'd say I think currently around like 30,000 or 40,000.
SPEAKER_00:  That's actually pretty amazing because each XP is around one minute of focus effort as far as I can tell. So to go through everything and now you can be very comfortable at an undergraduate level. Only 30,000 minutes, roughly speaking, is not that bad. That's pretty good.
SPEAKER_04:  Yeah. Oh, I should say just to make a distinction. Yes. That includes all of the content that is currently out. But if you are just wanting to prepare for undergrad level math, and you're not going to take linear algebra, multi-variable calculator, you're just taking foundations one, foundations two, foundations three, and methods approved. You're not in the traditional school sequence where in the traditional school sequence, they also cover a lot of material that doesn't actually get used in undergrad math. Just to name one off the top of my head, there's a bunch of circle theorems in geometry, inscribed angles.
SPEAKER_00:  Remember saying how much write about that on Reddit? Why you guys excluded that stuff?
SPEAKER_04:  Yeah, yeah, exactly. So we ripped out some of this. When we were building our foundation sequence to be the most streamlined, efficient way to get people up to university level math, we ripped down a bunch of that content that is not essential. It does not really come up in university level math. If somebody wants to, if somebody were to start out at the entry point of mathematical foundations one, which is like adding fractions, and they were to complete foundations one, two, three, I think, right, that is about 15,000 XB. And then plus the methods approved course, but this is actually kind of a relatively small course. I think it's something, I want to say something like 3,000 XB. It's like it's roughly like half or two thirds of size in one of those math foundation courses.
SPEAKER_01:  I think maybe you posted something where there was a student going at like a normal pace, had finished like 90% of it after six weeks or something. It seemed like a very approachable amount of content.
SPEAKER_04:  Yeah, yeah, yeah, yeah. Yeah, that is a proof. The amount of learning that happens per unit time, even per unit XB is just ridiculous.
SPEAKER_01:  It's funny that such a small amount of learning would probably resolve a lot of pain for so many people at university.
SPEAKER_04:  Yeah, exactly. Yeah, it's just this missing chunk of knowledge. So just to top it off, so that the 15,000 XB, let's say that's to get through foundations one, two, and three plus the methods approved. Let's say we're up to 18,000 XB. And let's just divide by 52 weeks. Let's say how much would this take you if you wanted to get up from adding fractions to completing methods of proof, knowing the essentials of calculus being like prepared for an undergrad math degree. Let's take 18,000 divided by 52 weeks in a year. We get to about 346. Let's say 350 XP a week. So 50 XP a day. Just do that for one year. You go from fractions, adding fractions, to ready for an undergrad math degree. That's wild.
None:  Yeah.
SPEAKER_00:  That is both an amazing comment on math academy at a terrible and timely education system.
SPEAKER_04:  That's the truth.
SPEAKER_01:  Simultaneously amazing and quite depressing.
SPEAKER_00:  Yeah, exactly. All right. Well, let's wrap it there. Two and a half hours of what I think is incredibly interesting information about this and incredibly interesting details. So thank you. Thank you very much for being here with us and for making the tool. And my thanks as well to Alex and Sandy and Robert and Elle or Jason and whoever else is involved. So seriously, thank you.
SPEAKER_01:  It's been absolutely wonderful, Jason. It's been awesome.
SPEAKER_04:  Yeah. It's great to have to meet both of you, Sandra, James. Hey, any more questions you have in the future? I'd be happy to talk more about it.
SPEAKER_01:  Awesome. We can definitely do it. If we've got plenty more to ask you to say, yeah? Awesome.
SPEAKER_02:  All right.
None:  Cool. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right. All right.