How Is Magic Pursuing Reliable AGI Systems?

TL;DR

General-domain, long-horizon reliability may be the last major obstacle to AGI, and solving it could require inference-time computation that concentrates vastly more resources on difficult decisions. Magic is pursuing this goal through proprietary large models, extremely long context windows, self-managing agents, and an AI software engineer designed to function more like a colleague than a simple application.

Transcript

thing that remains to be solved is uh General domain uh long Horizon reliability and I think you need in France time comput test I'm confused for that when you try to prove a new theorem in math or when you're writing a large um software program or when you're writing an essay of reasonable complexity you usually wouldn't write a token by token uh ... Read More

Key Insights

• General-domain, long-horizon reliability is presented as the remaining major AGI problem. Systems must reliably complete extended tasks such as proving mathematical theorems, writing large software programs, and composing complex essays, rather than merely producing plausible outputs one token at a time.
• Inference-time computation is described as essential for difficult reasoning. Steinberger argues that important tokens may require far more than 1X, 2X, or 10X the usual resources, potentially reaching a million times the computation if that effort can be used productively.
• Steinberger’s commitment to AI began at age 14. After exploring physics, mathematics, biology, and medicine, he decided that building a computer system able to handle many valuable problems could resolve his uncertainty about which individual field to pursue.
• Rigorous external criticism accelerated Steinberger’s development as a researcher. He asked Johannes Heinrich to review and challenge his work approximately every two weeks for a year, creating an intense, compressed research-training experience while Steinberger was still around high-school age.
• Research quality depends heavily on choosing consequential problems and improving solutions consistently. Steinberger credits Noam Brown with making early, non-obvious bets, selecting strong research targets, working extremely hard, and repeatedly refining ideas so progress compounds over time.
• Sustained focus has practical limits, according to Steinberger’s experience. While simultaneously studying, researching at FAIR, and operating Climate Science, he concluded that he could perform two major commitments well but not three, leading him to leave university.
• Magic was founded after Steinberger revised his estimate of AGI’s arrival. He had considered AGI much farther away when starting Climate Science, but transferred leadership of the nonprofit and returned to his original focus after deciding the timeline was significantly shorter.
• Magic’s strategy combines proprietary large models with long context and greater agent autonomy. The company aims to build an AI software engineer, argues that value will not concentrate in the application layer, and expects the strongest agents to manage themselves.

Questions & Answers

Q: What problem does Magic consider the main barrier to AGI?

General-domain, long-horizon reliability is presented as the main remaining barrier to AGI. A capable system must sustain accurate, useful work across extended tasks, including proving a new mathematical theorem, writing a large software program, or producing a reasonably complex essay. The challenge is not simply generating each next token, but allocating enough thought and computation to the decisions that determine whether the complete result succeeds.

Q: Why might AGI require more inference-time computation?

Inference-time computation allows a system to spend additional resources on especially difficult parts of a task. Steinberger argues that people do not normally produce a theorem, large program, or complex essay mechanically, one token at a time. Some decisions require concentrated thought. He therefore considers it important to find productive methods for spending not merely 1X, 2X, or 10X the resources on a token, but potentially a million times more.

Q: How did Eric Steinberger become interested in AI?

Eric Steinberger became interested in AI at age 14 while searching for work that would be meaningful and useful to humanity. He spent about a year considering physics, mathematics, biology, and medicine before encountering the idea of artificial intelligence. Building one computer system that could address many other problems appealed to him because it resolved his uncertainty about choosing a single field and gave him a clear long-term direction.

Q: How did Eric Steinberger train himself to conduct AI research?

Steinberger first learned programming because he wanted to solve AI problems, not because he was generally fascinated by computers. After working independently for several years, he contacted Johannes Heinrich, who had studied under David Silver, and proposed a demanding mentorship. Approximately every two weeks for a year, Heinrich criticized Steinberger’s work against a high standard while Steinberger tried to improve upon an algorithm related to Heinrich’s doctoral research.

Q: How did Eric Steinberger begin collaborating with Noam Brown?

Steinberger published work called single deep counterfactual regret minimization after spending years developing it. He said it performed slightly better than Noam Brown’s related deep counterfactual regret minimization work, although Brown had completed his paper much faster. Their ideas overlapped enough that Brown contacted Steinberger after publication. Steinberger then worked with Brown for about two years and conducted research part time at FAIR while studying at university.

Q: What research qualities does Eric Steinberger admire in Noam Brown?

Steinberger identifies problem selection, persistence, early conviction, intelligence, and intense effort as central strengths in Noam Brown’s research. Brown chooses important problems, makes bets before their value is obvious, and attacks those problems consistently so improvements compound. Steinberger also recalls highly efficient brainstorming sessions in which a problem that might otherwise have initiated six months of research could be clarified through a focused discussion with Brown.

Q: Why did Eric Steinberger leave Climate Science to start Magic?

Steinberger started Climate Science when he believed AGI was still much farther away. He cared about climate change and viewed nonprofit work as a valuable way to help the world, but AI had always been his primary calling. After concluding in 2022 that AGI was closer than he had expected, he handed over leadership of Climate Science and returned his attention to the AGI research agenda by founding Magic.

Q: What is counterintuitive about Magic’s approach to AI?

Magic’s approach includes training proprietary large models even while competing with much larger organizations. The company also argues that value will not accrue primarily in the application layer and that the best agents will eventually manage themselves. Its technical direction emphasizes an AI software engineer that feels like a colleague, very long context windows, and evaluation through work such as the open-sourced HashHop test.

Summary & Key Takeaways

• Eric Steinberger became obsessed with AI at 14 after considering physics, mathematics, biology, and medicine as possible ways to help humanity. Although mathematics came naturally to him, he chose AI because it appeared both suited to his abilities and more broadly useful, giving him a clear long-term direction for his work.

• As a young researcher, Steinberger sought rigorous criticism from Johannes Heinrich and spent years improving an approach related to deep counterfactual regret minimization. His result slightly surpassed Noam Brown’s related work, prompting Brown to contact him. Steinberger then collaborated with Brown at FAIR while attending university and also helped build Climate Science.

• Steinberger founded Magic after concluding in 2022 that AGI was closer than he had previously believed. Magic aims to automate software engineering and pursues proprietary large models, exceptionally long context windows, and agents capable of managing themselves. Its broader research target is reliable performance on complex, general-domain tasks extending across long horizons.