My Next Big Modeling Project: Behavior Coaching

by Justin Skycak (@justinskycak) on

Even if students are working on exactly the right things, they need to be working exactly the right way to capture the most learning from their time spent working.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

If you want an AI system to emulate the decisions of a human expert, then you need to give it all the key information that the human uses during their decision-making process.

That’s been our philosophy at Math Academy, and we’re about to start taking it to a whole new level.

Currently, we have our curriculum organized into a gigantic, super information-rich knowledge graph of

  • thousands of topics,
  • tens of thousands of knowledge points, and
  • tens of thousands of "edge" relationships between them (e.g., A is a prerequisite of B, but also other kinds of relationships).

And to determine the most optimal tasks for a student to work on, we

  1. overlay every single one of their answers onto the knowledge graph,
  2. have the information "flow" through the graph,
  3. determine the student's mastery & memory state at each topic,
  4. identify topics in need of review (sufficiently low memory state) and topics on the student's "knowledge frontier" (not yet learned but all prerequisites mastered), and
  5. compress those into the smallest set of tasks that "knocks out" all the due review while simultaneously maximizing the rate at which the student learns new material and pushes their knowledge frontier forward.

More details here, here, and here.

Suffice to say, this process works great and it’s enabled us to drastically speed up a student’s pace of learning by charting an efficient path through the curriculum that is tailored to their needs!

However, there’s another piece of the puzzle that’s still needed: behavior coaching.

We get the student learning/reviewing exactly the right topics, working on exactly the right exercises – but we still need to get the student engaging in exactly the right behaviors that are productive for learning.

In other words: even if students are working on exactly the right things, they need to be working exactly the right way to capture the most learning from their time spent working.

The most obvious example is that when a student works on math, they should be focusing fully on the math! They shouldn’t be continually getting distracted and interrupting their train of thought.

(Interruptions have an outsized negative effect in the context of deliberate practice: when you’re practicing at the edge of your ability, 30 minutes of full-focus practice will move the needle much further than 60 minutes of half-focus where your train of thought is continually being interrupted.)

However, focus is just one example of numerous behaviors that are necessary to support productive learning – and these other behaviors can be even more difficult for students to manage on their own because they tend to be subtle and involve balancing a tradeoff between two extremes. For instance:

  • Read the instructional material carefully before you attempt to solve the problems -- but don't spend too much time on it, because the problem-solving is where the learning happens!
  • When you solve a problem, try to minimize your reliance on reference material. Instead, retrieve information from memory. But if you're really stuck and can't remember what to do despite your best effort to recall, don't just sit there! Peek back at the worked example to refresh your memory. But then try to solve the rest of the problem unassisted -- don't solve the problem side-by-side with the worked example!

Now, I could go on for a while listing micro-behaviors that are necessary to support learning…

But even with a fully comprehensive list of micro-behaviors, few students in need of it would actually read it and put it into practice.

What’s needed is just-in-time feedback: detect when a student is engaging in unproductive behavior and tell them how to fix it.

(And incentivize them to fix it – so that regardless of whether the student is intrinsically interested in understanding the material deeply, or they just want to earn enough XP to keep their parent/teacher off their back, the quickest path to reward is to work productively and maximize their learning.)

That’s my next big modeling project.


Want to get notified about new posts? Join the mailing list and follow on X/Twitter.