Dwarkesh Patel on AI Research, AGI Careers & The Scaling Era

TL;DR

Making novel cross-field connections likely requires significant reinforcement learning, not just pre-training, because pre-training gives flexible general knowledge but not the research skill of discovery. Current LLMs behave like idiot savants with primitive memory scaffolding, memorizing exact text yet failing to generalize the way humans do.

Transcript

Today, this is going to be an Ask  Me Anything episode. I'm joined   by my friends Trenton Bricken and Sholto  Douglas. You guys do some AI stuff, right? Yeah. We dabble. They're researchers at Anthropic. Other news; I have a book launching today, it's called The  Scaling Era. I hope one of the questions ends up being why you should buy this book. ... Read More

Key Insights

• The Scaling Era is a curated book from Stripe Press compiling the most insightful snippets across Dwarkesh Patel's interviews with AI lab CEOs, researchers, economists, and philosophers, sliced by topic across conversations for a digestible read.
• LLMs memorize all of human knowledge yet rarely make novel connections across fields, unlike humans who occasionally link disparate facts, such as noticing that magnesium deficiency's effect on the brain mirrors migraine structure.
• The pre-training objective imbues flexible general knowledge about the world but does not necessarily imbue the skill of making novel connections or research, which people acquire through PhD programs and interacting with the world.
• Significant reinforcement learning on similar tasks is likely the minimum needed for models to approach making novel scientific discoveries, and the field has not yet done this in a meaningful or scaled way.
• Memory scaffolding for models is very primitive right now; models cannot construct summaries to retain new lessons the way a human learner deliberately would, and most training is just predicting the next word.
• LLMs may be idiot savants, comparable to Kim Peek who had encyclopedic memory but social debilitations, being amazingly good at niche topics while totally failing at others.
• Perfect, unforgettable memory can be debilitating, like a transformer context window of trillions of tokens where attending to every past detail prevents extracting generalizable insights.
• Humans learn best as children yet have total amnesia of childhood, while LLMs sit at the opposite end, capturing exact Wiki text phrasing but failing to generalize in obvious ways.

Questions & Answers

Q: What is The Scaling Era book about?

The Scaling Era: An Oral History of AI, 2019-2025 is a Stripe Press book by Dwarkesh Patel that compiles and curates the best, most insightful snippets from years of his interviews with AI lab CEOs, researchers, economists, and philosophers. It is sliced by topic across interviews, letting readers move page by page between perspectives, such as Dario on why scaling works, Demis on DeepMind's plans, and technical explanations of how models work.

Q: Why should ordinary people care about the Scaling Era book?

Dwarkesh frames it as a distillation of many fields of human knowledge applied to the most important questions humanity faces, arguing AI is one of the most multi-disciplinary fields imaginable. The book addresses the fundamental nature of intelligence, what happens with billions of extra workers, and how to think about an intelligence greater than the rest of humanity combined, making it accessible even to non-experts like the hosts' parents.

Q: Why can't LLMs make novel connections across different fields?

Sholto suggests the pre-training objective imbues flexible general knowledge about the world but does not necessarily imbue the skill of making novel connections or research, which people develop through PhD programs and interacting with the world. He believes at minimum you need significant reinforcement learning in similar tasks for models to approach making novel discoveries, and the field has not done this in a meaningful or scaled way yet.

Q: How do humans differ from LLMs in making knowledge connections?

Humans occasionally link disparate facts into discoveries, such as the example of someone noticing that a brain after magnesium deficiency has exactly the structure seen during a migraine, leading to magnesium supplements curing migraines. Scott Alexander notes humans also lack logical omniscience and do not think through every possible connection, but the hosts stress humans have demonstrably made such discoveries while they know of no example of an LLM ever doing it.

Q: What is the memory limitation problem with current AI models?

Trenton wonders if models simply aren't good at knowing what memories to store, since most training is predicting the next word and remembering specific facts. A human learning something new would deliberately construct a summary that sticks, but models currently lack that opportunity. He describes memory scaffolding in general as very primitive right now, citing Claude Plays Pokemon where iterating on the memory scaffold quickly improved performance.

Q: Why might LLMs be considered idiot savants?

Trenton uses the analogy of Kim Peek, who was born without a corpus callosum so each brain hemisphere operated independently, reading two pages at once with perfect encyclopedic memory but suffering other debilitations like functioning socially. Similarly, LLMs are amazingly good at very niche topics yet can totally fail at others, suggesting a savant-like profile of narrow brilliance combined with surprising gaps in capability.

Q: Why can perfect memory be debilitating for learning?

The hosts cite a case study of someone with perfect memory who never forgot anything, but whose memory was too debilitating. The analogy is a transformer whose context window is trillions of tokens, where you spend all your time attending to past things and become too trapped in the details to extract any meaningful, generalizable insights. Some forgetting appears necessary to generalize well.

Q: How does human forgetting relate to how children learn?

Referencing Terrence Deacon's book, the hosts note humans learn best as children yet forget literally everything from childhood, experiencing total amnesia of that period. Adults occupy an in-between state where they don't remember exact details but can still learn decently. LLMs sit at the opposite end of this gradient, capturing the exact phrasing of Wiki text yet failing to generalize in obvious ways, echoing Gwern's optimizer theory.

Summary & Key Takeaways

• Dwarkesh Patel announces his new book, The Scaling Era: An Oral History of AI 2019-2025, made with Stripe Press. It compiles curated snippets from his interviews with lab CEOs, researchers, economists, and philosophers, addressing questions like the nature of intelligence and the economics of billions of extra workers.

• A listener question raises why LLMs, despite memorizing all human knowledge, fail to make cross-field connections. Scott Alexander notes humans also lack logical omniscience, but the hosts emphasize humans have demonstrably made such discoveries while no LLM example is known.

• Sholto argues novel discovery needs significant RL beyond pre-training, since pre-training gives general knowledge but not research skill. Trenton points to primitive memory scaffolding and the idiot-savant analogy of Kim Peek, plus the trade-off between perfect memory and useful generalization.