How Does a 7M-Parameter Model Solve Hard Puzzles?

TL;DR

A 7 million parameter Tiny Recursive Model improves difficult puzzle solutions by repeatedly updating its current answer and the reasoning trace behind it. Its two-layer design favors recursion over network depth, reaching 45% accuracy on ARC-AGI-1, 8% on ARC-AGI-2, 87% on Sudoku Extreme, and 85% on Maze Hard while using a single tiny network.

Transcript

This 7 million parameter model that is million with an M is beating some of the top frontier models out there at the hardest reasoning benchmarks. It has already defeated Gemini 2.5 Pro, Deepseek, 03 Mini, and others. And it is a fraction of their size using a new approach that has taken everybody by surprise. I'm going to break it all down for you... Read More

Key Insights

• The Tiny Recursive Model is a 7 million parameter reasoning system that achieved 45% test accuracy on ARC-AGI-1 and 8% on ARC-AGI-2. The transcript says these results exceed those of several much larger frontier models while using less than 0.01% of their parameters.
• Autoregressive language models are vulnerable to compounding errors because they construct answers by predicting the next token. A single incorrect token can invalidate a solution, especially on difficult question-answering and structured puzzle tasks that demand precise, internally consistent outputs.
• Chain-of-thought reasoning improves reliability by generating step-by-step reasoning traces before producing a final answer. However, the technique is expensive, depends on high-quality reasoning data, and remains brittle because an incorrect generated reasoning path can still lead the model toward a wrong conclusion.
• Pass-at-K works by generating multiple candidate responses and selecting one that is expected to be accurate. The transcript presents it as another test-time strategy, but argues that sampling more next-token predictions does not by itself resolve the underlying limitation of autoregressive reasoning.
• Deep supervision improves an answer through multiple supervised refinement stages while carrying latent features forward as initialization. In the earlier HRM approach, deep supervision doubled accuracy from 19% to 39%, whereas recursive hierarchical reasoning produced a smaller improvement from 35.7% to 39%.
• The Tiny Recursive Model simplifies HRM by using one small network instead of two medium-sized networks. It discards the hierarchy, biological analogies, and complex mathematical justification, preserving recursion as the central mechanism that repeatedly improves a proposed solution.
• The model maintains two memories: its current answer and the reasoning trace used to reach that answer. Each recursion updates both memories, resembling repeated Sudoku attempts in which the system evaluates what worked, adjusts its reasoning, and tries an improved move.
• Two layers produced better generalization than deeper versions of the Tiny Recursive Model because adding layers increased overfitting. The researchers instead reduced network depth and proportionally increased recursion, using repeated computation to provide virtual depth while retaining a very small physical network.

Questions & Answers

Q: What is the Tiny Recursive Model?

The Tiny Recursive Model, or TRM, is a reasoning system containing only 7 million parameters and a single tiny network with two layers. It solves difficult structured problems through repeated refinement. During each recursion, it updates both its current proposed answer and the reasoning trace that led to that answer, allowing later cycles to correct or improve earlier attempts.

Q: How does the Tiny Recursive Model perform recursive reasoning?

The Tiny Recursive Model performs reasoning through a feedback loop that repeatedly updates two memories: the current guess and the reasoning trace behind that guess. Each cycle uses the existing state as the starting point for another improvement. The transcript compares this process to trying a Sudoku move, evaluating why it worked or failed, adjusting the approach, and trying again.

Q: Why does the Tiny Recursive Model use only two layers?

The Tiny Recursive Model uses two layers because experiments described in the transcript found that adding more layers reduced generalization through overfitting. The researchers obtained better results by decreasing network depth while proportionally increasing the number of recursive loops. Two layers were reported as the optimal choice, with recursion supplying virtual depth through repeated use of the same small network.

Q: How is the Tiny Recursive Model different from HRM?

TRM replaces the Hierarchical Reasoning Model's two medium-sized networks with one tiny network. It removes the proposed hierarchy, biological arguments, and complex theoretical justification while preserving recursion. Instead of describing two latent features as brain-like systems operating at different speeds, TRM treats them more directly as the current answer and the reasoning trace that produced it.

Q: What results did the Tiny Recursive Model achieve?

The Tiny Recursive Model reached 87% accuracy on Sudoku Extreme, 85% on Maze Hard, 45% test accuracy on ARC-AGI-1, and 8% on ARC-AGI-2. Compared with the earlier HRM approach, those figures improved from 55%, 75%, 40%, and 5%, respectively. The transcript emphasizes that the ARC-AGI-2 increase from 5% to 8% is meaningful despite appearing numerically small.

Q: Why do large language models struggle with hard reasoning tasks?

Large language models struggle with hard reasoning tasks because they generate responses autoregressively by predicting the next token. One incorrect token can make an entire answer invalid. Chain-of-thought and additional test-time computation can improve performance, but generated reasoning may still be wrong, and producing many reasoning tokens increases expense while requiring high-quality reasoning data.

Q: What role does deep supervision play in recursive reasoning?

Deep supervision breaks improvement into multiple supervised stages and carries latent features forward as initialization for the next stage. In the HRM results cited in the transcript, it doubled accuracy from 19% under single-step supervision to 39%. By comparison, recursive hierarchical reasoning increased accuracy from 35.7% to 39%, suggesting that repeated supervised refinement provided the larger contribution in that earlier system.

Q: Why is recursion useful in a small reasoning model?

Recursion lets a small network perform repeated rounds of computation without adding physical layers or greatly increasing its parameter count. Each loop revisits the proposed solution and its associated reasoning trace, enabling incremental correction. The transcript describes this repeated computation as virtual depth, which helps explain how a two-layer, 7 million parameter model can generalize well on Sudoku, maze, and ARC-AGI tasks.

Summary & Key Takeaways

• Tiny Recursive Model challenges the assumption that better reasoning always requires larger language models. With only 7 million parameters, it repeatedly refines solutions instead of relying primarily on autoregressive token generation, long chain-of-thought traces, or numerous sampled responses. The approach targets structured puzzles where a single incorrect prediction can invalidate an answer.

• The model simplifies the earlier Hierarchical Reasoning Model by retaining recursion while removing its two-network hierarchy, biological justification, and complex theoretical framing. It maintains two forms of memory: the current answer and the reasoning trace that produced it. Every recursive cycle updates both, creating a feedback loop for gradual correction and improvement.

• Experiments described in the transcript indicate that two layers generalize better than deeper alternatives because additional layers increased overfitting. Increasing recursion supplies virtual depth without expanding the physical network. Compared with HRM, TRM improved Sudoku Extreme accuracy from 55% to 87%, Maze Hard from 75% to 85%, and both ARC-AGI benchmarks.