Five Steps to Becoming a Fully-Fledged Quantitative Software Engineer

by Justin Skycak (@justinskycak) on

Once you get past steps 1-3, it's hard to find scaffolding. You can't just enroll in a course or pick up a textbook. The scaffolding comes from finding a mentor on a mission that you identify with and are well-suited to contribute to. And it can take a lot of searching to find that person and problem area that's the right fit.

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

The difference between creating toy projects vs production-quality systems… it’s a massive transition.

Here were the stages for me personally:

  1. learning basic coding
  2. doing projects scoped down to a toy research project or class project
  3. working as a data scientist, some semi-complicated analyses/models but not running in real time
  4. building a massively complex expert system that runs for a tiny number of users, quickly in real time
  5. getting that system really robust to all sorts of edge cases and data quality issues, running safely for a larger number of users

Steps 1-3, I was able to pull off mainly through self-guided learning. Steps 4-5, on the other hand… the guidance I’ve received from Jason Roberts has been invaluable.

Steps 1-3 can be done using structured resources (courses, textbooks, etc.) and self-directed toy projects. But eventually there comes a point of diminishing returns where you’ve maxed out your ability to do small-scale stuff.

That’s when you need to transition into large-scale stuff. The difference between steps 1-3 and steps 4-5 is scale and complexity. You need to sink your teeth into a really big, really complicated problem.

And that’s where it becomes super useful to have a mentor who helps you

  1. focus on a big unsolved problem that's worth solving, and
  2. chip away at that problem by solving a series of progressively larger subproblems.

Jason is that mentor for me. The best analogy I have for describing our relationship is that it’s like doing a super intense PhD under the world’s most demanding AND supportive advisor.

  • The advisor is demanding: they want not just a thesis but also a production-grade software product that is robust and relevant enough to successfully launch a company. And not only is that way more work, but they want it done way faster than the duration of time that normally constitutes a PhD.
  • But the advisor is also supportive: instead of meeting with you once a week, they'll chat with you at length every single day. They're not hedging their bets across a wide variety of grad students, they're going all-in on you. You're expected to deliver, but you're also given enough support that you have no excuse for not delivering.

Now, I should be clear: the support is pretty high-level. It’s not the kind of hand-holding that you might find in a class at school. And this is by necessity.

  • It's expected that you're coming in with enough foundational skills that if you're missing anything, your knowledge gaps should be small enough relative to your figure-it-out-itude that you can backfill them on your own with some high-level guidance. The mentor doesn't have time to school you.
  • It's even expected that you outskill the mentor in various places! You bring some superpowers to the table, and while the mentor doesn't know how those superpowers work at the lowest level, they're able to supply high-level guidance that corrals and focuses those superpowers in a way that puts serious leverage on the big problem that they're having you help them solve.

But the support is also ever-present. If you are pushing on the same needle as the mentor, and seriously moving that needle, and producing good ROI on the time that the mentor invests in you, then they will keep giving you their time and do whatever it takes to keep you moving that needle.

Here’s an concrete example of what I mean by that.

Jason and I started working on the automated task selection algorithm the summer of 2019, and when the pandemic hit in 2020, it was clear that we had to get the Math Academy platform fully automated to support remote learning for our Pasadena school program in the fall.

So you know what we did? Jason and Sandy invited me to move in with them to quarantine, which essentially led to a makeshift startup incubator experience run out of their living room. I worked with Jason pretty much every waking hour, well into the night, every day including weekends, so that by the end of summer we were ready to run entire school classes on the automated system.

Now, I’ve been incredibly lucky to have found a career mentor as demanding and supportive as Jason. (I should be clear: “demanding” is a desirable quality in a mentor when paired with a commensurate level of support.)

But the luck doesn’t happen if you’re not actively searching. In the 5-or-so years of my (semi) professional life before I met Jason, there were about 10 other potential mentors I worked with that I could have stuck around with for the long term. It took a lot of searching to find the right fit.

And I think that’s the reason why it’s so hard to go from steps 1-3 to steps 4-5.

Once you get past steps 1-3, it’s hard to find scaffolding. You can’t just enroll in a course or pick up a textbook. The scaffolding comes from finding a mentor on a mission that you identify with and are well-suited to contribute to. And it can take a lot of searching to find that person and problem area that’s the right fit.


Want to get notified about new posts? Join the mailing list and follow on X/Twitter.