An Interview Question for LLM Research Assistants

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.

Here’s the interview question that I’m planning to use as an initial sanity check to determine whether any LLM-based research assistant is even worth looking into:

“Summarize the research on how individual differences like learning styles and working memory capacity affect student learning rates and academic success, especially in the subject of mathematics.”

This is kind of a trick question because tailoring instruction to a student’s perceived learning style is not scientifically supported, even though there’s a lot of info online and in predatory journals advocating for it.

There is also lots of info online falsely claiming that all students learn at the same rate and any observed differences in student learning rate can be chalked up to differences in prior learning opportunities.

And, of course, working memory capacity varies between people and has been shown to impact learning rates / academic outcomes, but people often resist acknowledging that uncomfortable truth and will pivot the conversation into other things that are less relevant but more “feel-good” to talk about.

I’m excited by the prospect of speeding up literature searches, but at the same time, it’s a big step up in difficulty when the problem becomes not just summarizing information but also determining which sources are credible.

Results

Kit and Zander gave this prompt to OpenAI DeepResearch and shared their results. Below were my reactions:

x.com/justinskycak/status/1890965338759544852

x.com/justinskycak/status/1890975927158407224

Want to get notified about new posts? Join the mailing list and follow on X/Twitter.