MTS Interview: Riemann Hypothesis

by Justin Skycak on


Watch on YouTube


[0:58] Math used to get a “slam dunk” breakthrough every couple of decades. Now AI is dramatically accelerating the pace of mathematical discovery. From my position, it's like watching your favorite TV series, and you used to get one episode every ten years, and now there's a new episode every month. I love it.
[6:08] Not knowing how to use LLMs will soon be like not knowing how to open a spreadsheet.
[9:11] If your skill level used to be a 1 out of 10 and now with the AI you're like, “oh, I'm a 10 out of 10”... Guess what? The scale's out of 100 now. If you were a 10 out of 10, and you multiplied yourself 10x with AI, you became 100. And if you were a 0 out of 10, you're still a 0. All the leverage in the world cannot leverage a zero. That's going to be the lesson of the decade: Leverage isn't worth anything until you have something worth multiplying.
[14:16] Math Academy treats students' time like money on a trading floor: maximize the learning return on every minute invested.
[19:12] “You're Not Dumb, You Just Lack the Prerequisites” blog post by our student Lelouch.
[26:54] Don't just ask LLMs questions. Get them to ask YOU questions. Learning requires actively solving problems, not passively reading explanations.
[32:53] LLMs let you “move at the speed of thought.” But if your thinking is messy, you'll just produce bad code and compound problems faster.
[36:13] It's easy to doubt the value of expertise when you see AI producing incredible results like proving the Riemann hypothesis or Navier-Stokes. But you have to remember who's steering the models into these incredible results. It's not just random people off the street who are like, “Hey, Claude, prove the Riemann hypothesis. Oh, it proved it. Great.” That's not what's happening. Everywhere I look, the really impressive results of AI production are coming from people who have serious expertise in the domain they're applying AI to.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.


Watch on YouTube


The transcript below is provided with the following caveats:

  1. There may be occasional typos and light rephrasings. Typos can be introduced by process of converting audio to a raw word-for-word transcript, and light rephrasings can be introduced by the process of smoothing out natural speech patterns to be more readable via text.
  2. The transcript has been filtered to include my responses only. I do not wish to infringe on another speaker's content or quote them with the possibility of occasional typos and light rephrasings.
∗     ∗     ∗

[00:00:00]

Hey, Madison. It’s great. I know, me too. It’s been a while. It’s great to talk to you again. That’s exciting. Sounds great.

Riemann hypothesis. Yep, I saw that just a couple hours ago too. I was like, “What? Where’d that come from?” Right?

[00:01:00]

My pleasure. Thanks for having me on, Madison. This was great. It’s real exciting.

Well, I should say I use mathematics professionally, but I’m not a research mathematician. So my words don’t speak as a member of the research mathematician field. But as somebody who interfaces with a lot of math every day and studied a lot of math in college, et cetera, I’m really excited.

I mean, that’s the first thing that comes to mind. I see something like that drop, and I mean, the Navier-Stokes from a while back is just incredible.

My perception of [00:02:00] math has always been that it’s kind of… it’s a very slow-moving field because it’s just, it’s so…

Papers take years to get through review cycles, and then the results… If you’re a spectator watching the game of math play out, it’s like every couple decades maybe you get a slam dunk. At least, I’m sure there’s more that goes on internally in the field, but to people who are watching from the outside, it’s kind of like it takes a while for things to unfold.

And now just seeing this crazy rate of progress, Navier-Stokes, Riemann, I mean, I’m sure there’s more that has to be done in terms of consolidating the results into human-interpretable form. But it’s amazing.

Well, just watching from the outside, my impression has been it’s kinda like… [00:03:00] There’s a saying in software of, “Okay, make things work, then make them right.”

And for the longest time, it would take forever to make things work in math research, and now you can just get a lot of results really fast.

And at least my impression is that the field is kind of evolving into a sort of, “Okay, AI is gonna be here, LLMs are gonna be here for the foreseeable future. It’s gonna be a tool that’s incorporated into the mathematician’s toolkit.”

Like how mathematics used to be entirely by hand, right?

And then computers came along, and now there’s a lot of mathematicians who use code and stuff, and then that’s been normalized in the field.

And now we’ve got AI, LLMs. That’s gonna be normalized too.

But I think there’s gonna be a lot of emphasis now on kind of consolidating the [00:04:00] results into, “Okay, we’ve proven this, we’ve proven that. What have we learned mathematically from that?”

‘Cause my impression is a lot of these problems are set up with the intention of, in addition to just solving the problem, forcing some kind of really cool mathematical developments.

And so solving the problem is step one, and finding all the cool math that emerges from the solution techniques in there, I imagine, is the next step.

And at least that’s how it’s gone in my own work when I use AI, LLMs, for mathematical problems that I face on my own in coding, quantitative coding stuff.

That’s kinda how it goes often, is step one is just try to get some kind of model that behaves as desired and whatever, kind of in broad strokes, and then really consolidate [00:05:00] down into making it elegant and building it into a system.

At least that’s the way I see it. And the way I see it is there’s a lot more down this path. It’s an exciting path, and it’s like you saw, you take a step that people used to think would take a lot longer to take. It’s like that’s not the end of the journey.

There’s still so much more of the journey to do, and it’s exciting ‘cause it’s like watching a… I mean, from my position, it’s like watching a, imagine your favorite TV series, and you used to only get one episode every 10 years. And you’re like, “Man, that was a really good TV series. I wish that I could get one episode a year, or can we do monthly or biweekly?”

And now it’s every month, there’s a new episode. [00:06:00] It’s great. I love it.

I mean, I think that’s the question that a lot of people in general are asking right now, and a lot of people are debating over right now. It’s like, what’s the takeaway? Does it mean, is this just interesting to the field of mathematics? Are there other problems in applied math or just technology in general that depend on all this information, on all these theorems and whatever?

What does this mean for coding? And I think… I mean, there’s so many different views, a lot of them conflicting, but I think at the heart of it is, at least the way I see it, we just have another tool.

Another tool in the toolkit can do incredible things, and it’s worth learning regardless of [00:07:00] whether you’re a pure mathematician, an applied mathematician, a technologist, somebody who doesn’t interface with math at all.

You should probably learn LLMs. You should probably learn how to use them well ‘cause they’re… I imagine even several years from now, not being able to use an LLM is gonna be like not being able to open up a spreadsheet on the computer.

It’s like, “What are you doing?” [00:08:00]

See, the reason why I’m excited is ‘cause I have so many problems to solve. I have no shortage of… Every day I have 10 problems to solve, and I get through maybe one of them, and I’m just… It’s day after day.

And I think maybe it’s the difference of… I mean, we’re working in a startup with a lot of future growth ahead of us and just kind of in a growth phase and a scaling phase, and just every day everywhere is on fire, and you just have to choose what fires to put out.

So I’m more than happy. It’s not like I have one problem that I’ve been kind of nurturing for years and I’m like, “That’s my problem. Nobody touch it. It’s mine.”

I’m just like, “Let’s go. Just fewer problems. Solve problems is good.” [00:09:00] You solve one and three more appear.

It’s the Hydra effect. That’s what I’ve learned. I used to think that solving problems in my line of work would lead to calmer, less chaos.

But it’s like it just expands your opportunities. If you’re solving a good problem, at least in my experience, that results in so many subproblems being opened up, opportunities being opened up into the future.

I think this is one of those… [00:10:00] It’s easy to think that because a tool can do stuff for you, that it’s like, “Oh, it’s taken away from me the opportunity to do things.”

But I think in reality what’s happened is that the tool, if you have a lot of expertise, this tool can extend you in ways.

It’s a multiplier on you. And you get used to thinking, well, the bar used to be here, and now AI tools can basically do that. You don’t just walk into developing a product by not having… The level of expertise that you need to have, it used to be a level that [00:11:00] you could kind of use LLM tools for, but now it’s kind of like the bar has just been raised.

The multiplier has been raised. You wanna go do something serious out in the world, well, it’s just like previously when coding came to math, the advent of calculators and stuff, it’s like you use the calculator to extend your mathematical abilities to build stuff as an engineer.

Having a calculator doesn’t mean that you don’t need to know what you’re doing anymore. It means that when you know what you’re doing, you can do more of it faster and extend it.

And I think LLM is this very similar kind of setup, but just to software in general and domain expertise in general.

It’s like the more you know what you’re doing, the more you see the surface area of all these problems to solve with LLM, what problems are worth solving and what avenues you can explore with them.

So I would say it’s very much a multiplier, and if you have a large amount of baseline [00:12:00] expertise and you really know what you’re doing, then you’re gonna…

If your skill level used to be a one out of 10, and now with the AI it’s like, “Oh, I’m a 10 out of 10.” Well, guess what? The scale’s out of 100 now. If you were a 10 out of 10, you become 100. That’s basically my take on it.

[00:13:00]

If you don’t know the fundamentals of what you’re doing, you always gotta take other people’s word for it. And that’s when you kind of get into this trap of, “Oh, so-and-so said this is the best practice or the solution.” You don’t think about it yourself.

You just do what they do. Or an LLM said, “This is the right approach to that.” If you don’t have your baseline skills, you can’t evaluate whether that is the right approach or not, and you can’t get really off manifold from the LLM, kind of extend the base training set that it’s on.

And in my experience, so much of the value in producing software is kind of taking your domain expertise, that stuff that other people typically don’t know or don’t understand, other [00:14:00] models don’t know or don’t understand, and being able to wrangle code and math and just engineering in general to perform things that it wouldn’t normally be able to do.

And I just think without the foundational skills, it’s kind of like you’ve got a very low ceiling of what you’re able to… You’re gonna be led by the LLM instead of you leading the LLM.

We are a fully automated, adaptive, individualized learning platform. The most efficient way to learn math, hands down.

The whole platform is designed with the intent that every minute that you spend practicing, every second that you spend on the system, we’re trying to maximize your learning efficiency.

We take this [00:15:00] as seriously as almost like people have a trading floor. It’s like, okay, we’re paying in student time. Students are paying their time into the system, and we wanna maximize the return on your time because that is just such a big lever in education.

If you can convert student time into better results, more learning, then that opens up so many opportunities for students. And the way that we kind of approach this is really leveraging a lot of cognitive science, science of learning principles that have been around for decades but have never really found their way into software previously.

[00:16:00]

Basically, we got our curriculum, and the main course is a lot of high-level university math, but the idea is you place into it, and we’ll automatically find what are your missing prerequisites.

If you’re really rusty on math, it can go all the way down to, okay, maybe you’re not even ready to start…

We got a whole sequence of courses that goes up to there. You can take a diagnostic. If you have forgotten some basic algebra, it can hammer that in, and it kind of meets you where you’re at and builds you up to that level.

[00:17:00]

Sure. So I would say, pretty much, if we wanna talk about the principle components of effective learning, it is kinda like what are the main things that often don’t happen in student learning experiences that we have built software to make happen? Here’s the main components.

First is meeting you where you are in terms of whenever we teach you something on the system, we wanna make sure it’s something that you don’t know yet, you’re not solid on yet, but you are solid on the prerequisites for it. Right [00:18:00] at the edge of your knowledge frontier.

And typically, in a classroom or something, it’s just teacher teaches for the class as a whole, and even then often it’s not even really for the class as a whole, it’s just a calendar schedule. This is what we’re doing today, then the next day, then the next day.

Exactly. So you might be needing to do some integration by completing the square, maybe, an algebra technique, and maybe you’ve forgotten how to complete the square. It’s like, well, you’re not ready to layer on.

It’s almost like asking somebody to do a figure skating spin before they can even skate backwards or something.

It’s like let’s get you to the baseline. Just start where you are and build up from there. So that’s one component. That’s mastery learning, basically, is what it’s [00:19:00] called.

And actually, one of our students wrote an article called, I think it was, “You’re Not Dumb, You Just Lack the Prerequisites.” And that made rounds through the internet.

I remember two years ago I posted on Hacker News, and it just blew up and I…

It comes back ever since. People love it because it really hits on such a big emotional component of learning, is that exactly as you said, when you are asked to learn something and the gap between your level of skill and the thing that you’re being asked to learn is too big, then you’re gonna come away thinking that you [00:20:00] just don’t have the talent to learn it, when in reality, you may just be missing a lot of context.

And I love, in that article, “You’re Not Dumb, You Just Lack the Prerequisites,” our student, Lej Luch, gave this great analogy, which is, it’s like you come into a movie halfway through. You didn’t even watch the first half of the movie, and you’re confused.

And do you berate yourself for being confused? You’re like, no. I didn’t even see the first half. Of course I’m confused. Why should it be surprising that I’m confused? It’s not a character failing. It’s just that I’m missing context. And same thing in learning.

[00:21:00]

Absolutely. Spaced repetition is just every time you recall a piece of information… Well, let’s step back. After you learn a piece of information, you start forgetting it.

You got this decay curve of your memory. You can’t just look at something once and then remember it forever.

That’d be great, but that’s just not how the brain works, and it’s actually kind of efficient that way and then forces you to remember things that are actually important, that you use a lot.

Anyway, but the end result of all this neurobiology is that you look at something, you consume a piece of information, you don’t have it locked in, you are forgetting it.

And you have to practice retrieving that information from memory in order to slow the rate of forgetting.

So spaced repetition is a system for slowing your rate of forgetting. Maybe you learn something today, you practice recalling it, all of it, then you practice recalling it [00:22:00] again maybe tomorrow or a few days, then a week later, a couple weeks later.

Every time you practice recalling it and you’re successful, it extends your memory duration. Your rate of forgetting slows down, so you can wait longer.

And if you do this, you can go for years having remembered information.

And now normally people think of spaced repetition in terms of flashcards, vocab words, definitions, stuff like that.

And that, for… That’s kind of the familiar interpretation of it, it’s a medical student studying anatomy or whatever. Medical students love spaced repetition because of all the information they have to get in their brain.

Now, we have adapted spaced repetition for mathematics in this actual skill hierarchy.

So not just [00:23:00] a bunch of independent flashcards, but actually solving math problems according to a spaced repetition schedule.

And so maybe you solve an algebra problem today. You learn how to solve an algebra problem today. You practice solving it a bit, and then a few days later, a week later, a month later, you’re gonna practice that same thing so you’re not forgetting it.

But one kind of particular consideration that comes up when adapting spaced repetition to math is when you are solving math problems, you’re often practicing a bunch of skills implicitly.

It’s like how if you’re playing basketball and stuff, you’re dribbling the whole time.

You’re practicing dribbling while simultaneously practicing running a play and passing and stuff. There’s a lot going on at once. And in math, this is a similar way.

In algebra, maybe [00:24:00] you’re practicing your multiplication, your division, as you solve an algebraic equation.

And maybe, as you… If you solve a quadratic equation, you kind of factor it, and then you have to solve two linear equations that pop out of that. So you’re implicitly practicing some simpler skills.

Happens with calculus too. Calculus, the joke, the running joke is calculus is really where students learn algebra ‘cause it forces so much algebra.

If you’re not solid on your algebra, you really struggle with calculus if you’re not solid on the algebra. So we track all of this.

[00:25:00]

Absolutely. It’s like you gotta really practice recalling and pulling information out of your head, not just putting it in there. Totally agree.

That’s right, not random flashcards. And problem design. And there’s so much that comes down to maximizing the [00:26:00] efficiency and coverage of what you’re reviewing, ‘cause it’s like you wanna…

On one hand, if information is important enough to practice recalling it, you wanna practice recalling it.

But also, you want to try to consolidate that information into as few recall events as possible.

So in math, for example, it’s like once you know your algebra really well, you don’t have to continually hammer every single algebra skill independently.

You can kind of consolidate it into higher-level problems that exercise all of this.

And you can do the same kind of stuff in other knowledge domains too, to where it’s like synthesizing material, pulling a lot of skills together, and you can implicitly review large swaths of knowledge with single skill retrieval events.

[00:27:00]

My first piece of practical advice for using an LLM to help you learn faster, or for using any resource to help you learn faster, is if it’s an off-the-shelf thing where you are kind of directing your own learning experience, you’re reading a textbook or using an LLM or anything, it’s like spin up on the fundamentals of learning.

There’s a lot of things that you can do while reading a textbook or using an [00:28:00] LLM that, if you don’t really know what you’re doing, you can come out of it with not much learning.

And we just talked about recall events.

I’ve seen people try to learn math by reading textbooks and not solving the problems. And they think that they’re learning a lot about… And they may be becoming a little familiar with some pieces of math, or there’s words that they can recognize and stuff, but really you don’t learn it very well unless you actually solve the problems.

Same as for LLMs. It’s like if you want to learn something from an LLM, it’s not just asking the LLM questions and having it answer your questions.

You should get it asking you questions, and you should have to answer the questions. A lot of fundamentals for this kind of thing.

I think it just again kind of comes down to learning the… [00:29:00] If you have basic foundational knowledge of the thing that you’re trying to do, you can use LLMs in an incredible, incredible way. But if you don’t, you can be led astray by it and think that you’re learning something when you’re not really.

One thing I would say too is, in the way that we leverage AI at Math Academy, we actually built our system between 2019 and 2022. That was really the brunt of the software development. This was pre-ChatGPT, and the AI that we kind of built into our system is actually an old-school expert system.

And the idea [00:30:00] being that it’s not a student talking to a chat for the session. It’s the mind of an expert tutor selecting the content that you’re gonna go through and solve problems on.

And the state of learning in general and application of learning science principles, most educational establishments and products don’t use any of the mastery learning, spaced repetition, interleaving, all that kind of established learning science through the decades.

And you don’t actually need an LLM to start applying this. LLM is definitely a multiplier, but the core foundations do not depend on having an LLM.

[00:31:00]

That’s true. It is kind of interesting seeing LLMs being such a powerful tool has simultaneously made a lot of people lazy, right?

It’s [00:32:00] like just hand the problem to the LLM, and it’s like really, if you wanna get the most bang for buck out of the LLM, it’s not just solving the problem, it’s also understanding what you’re doing.

Why were you not able to solve the problem yourself? And what are the steps, prerequisite pieces of prerequisite knowledge, pieces of context that would help you better wrap your head around problems of this class?

And if you really don’t take the lazy way out and just delegate entirely to the LLM, if you use it as a kind of partner, almost bring it in as a consultant where it’s like, “How would you solve this problem?

Okay. That’s really interesting. Sounds like you have a good solution. Seems like it works. Teach me how to… Teach me what you’re doing. I wanna…”

You kind of bring in, try to learn the skills of this outside consultant, and the more you know of these skills, the better you can direct this tool in the future.

[00:33:00]

Well, LLMs can generate lots of information, lots of code, lots of everything. It’s like you’re moving at… You can basically move at the speed of thought, right?

If your thinking is messy, though, if you don’t really know what you’re doing, you can just create lots of worthless stuff, or lots of stuff that you may have to undo later.

And that’s… I mean, this is another area where it pays off to really know what you’re doing. As a software engineer, if you know how to consolidate [00:34:00] complexity, if you have dealt with codebases, having to refactor stuff, having to… You’ve seen what happens when a codebase just gets sprawling, and then the compounding of bugs, and every time you try to build something on the system, then it creates this combinatorial explosion of pipelines and everything, and you’re just losing your mind, and it drags you down.

If you know how to consolidate that, then you can use the LLM as a power tool to help you move faster. But if you don’t really know what you’re doing and you’re just delegating everything, every little decision to the LLM, you’re just gonna compound your codebase into a thing that ultimately, it’s gonna take even longer to clean up later.

[00:35:00]

It is kind of funny. I think the thought experiment that really puts this in perspective for a lot of people is, imagine, just think about anything that you know a lot about, and then think about somebody who’s a beginner in that area.

And you think about how maybe they’re trying to do something, maybe it’s music, they know an instrument, they’re trying to play some songs, or maybe it’s writing about some topic that you know a lot of.

If you know a lot of this, then you would look at what they’re doing and just be like, “Nah, [00:36:00] I would do this, do that. Just do this instead. That’s gonna work better.”

And it’s the same way, I think, with modern LLMs. It’s easy to forget what multiplier your expertise is on these kind of tools.

[00:37:00]

I mean, I would say just, I guess a continuation of what I was mentioning earlier is, just think about… It’s easy to doubt the value of expertise when you see models doing crazy results like proving the Riemann hypothesis or Navier-Stokes.

But I think the thing to think about in that situation, all these crazy results that are coming out of these AI labs, these are not just random people off the street who are like, “Hey, Claude, prove the Riemann hypothesis. Oh, it proved it. Great.” That’s not what happened.

It took… The people behind all these AI-assisted proofs of [00:38:00] math theorems, they are mathematicians. For the most part, they are research mathematicians themselves, and if not research mathematicians, they are very, very well-practiced in mathematics. They actually know what they’re doing.

And everywhere I look, the most impressive results of AI production have come from people who have gotten manual reps practicing with the thing that they’re doing, who have kind of really tried to explore the space of their manual skill set, tried to…

They’re using the AI to their best ability, but they’re also trying to consolidate as much of whatever domain they’re in into their own head, so they can better guide this.

So I think it’s just kinda like whatever, [00:39:00] as a beginner, if you look at some incredible results coming out of AI, just ask yourself, “Who’s getting those incredible results that are making you…”

It’s not some random person off the street.

It was great to talk to you, Madison. Thanks for having me on.

Prompt

The following prompt was used to generate this transcript.

You are a grammar cleaner. All you do is clean grammar, remove single filler words such as “yeah” and “like” and “so”, remove any phrases that are repeated consecutively verbatim, and make short paragraphs separated by empty lines. Do not change any word choice, or leave any information out. Do not summarize or change phrasing. Please clean the attached text. It should be almost exactly verbatim. Keep all the original phrasing. Do not censor.

I manually ran this on each segment of a couple thousand characters of text from the original transcript.



Want to get notified about new posts? Join the mailing list and follow on X/Twitter.