How Does AI Model Routing Cut Agent Costs?

TL;DR

Keep a frontier model responsible for planning and difficult decisions, then delegate implementation and exploration to cheaper models. Cognition reports this structure cuts the cost of Fable-level intelligence by 40% while enabling deeper codebase exploration, but routing must account for changing agent tasks, uneven model strengths, token usage, cache behavior, and costly failure loops.

Transcript

[music] >> have been really exciting. We've tried to get a bunch of the industry leaders together to talk about some of the problems that are that we're facing as we try to run more on local. If you guys were here for the first panel, one of the things that we talked about was model routing. We firmly believe that we're in a multi-model world. I th... Read More

Key Insights

• Model routing is the practice of assigning planning, implementation, or specialized subtasks to different models according to their capabilities, efficiency, and cost. The panel presents it as an emerging discipline without a single established solution for production or local AI deployments.
• Cognition's routing design keeps a frontier model responsible for planning and difficult decisions while delegating implementation to cheaper models. This preserves frontier-level judgment instead of forcing users to begin with a weaker model that may fail and require an expensive switch.
• Cognition reports that delegation reduces the cost of Fable-level intelligence by 40%. Cheaper implementation models can consume more tokens or launch three subagents to explore a codebase more comprehensively while remaining within the budget associated with frontier-model execution.
• Model capabilities are jagged across tasks and subdomains because models are trained and post-trained on different corpora, teachers, and subtasks. A model that leads a general coding benchmark is not necessarily superior at every library, visualization task, modeling task, or workflow stage.
• Complementary model strengths can produce higher aggregate accuracy when a router assigns subtasks appropriately. The panel states that routing techniques can deliver up to 10% higher accuracy on LM Router Bench, although results depend on the available model pool and task.
• Initial task-type routing is fragile for agentic work because the task can change during a session. A request may begin as codebase explanation, progress to feature implementation, and end with live testing and debugging, leaving the initially selected model poorly matched.
• Token price alone is an incomplete measure of efficiency because weaker models may use substantially more tokens. The description notes that Haiku can thrash outside its training distribution, repeatedly call tools, and eventually cost more than the more expensive model would have cost.
• Context management affects both cost and intelligence because continuous context can keep the KV cache warm, with cached tokens costing roughly ten times less. Compaction creates a cache miss and can reduce quality, so Cognition treats compaction primarily as an intelligence decision rather than a cost tactic.

Questions & Answers

Q: How does AI model routing reduce agent costs?

AI model routing reduces costs by reserving a frontier model for planning and difficult decisions while assigning implementation and exploration to cheaper models. Cognition reports that this structure lowers the cost of Fable-level intelligence by 40%. Because cheaper models cost less per token, they can perform more extensive work, including sending three subagents to examine a codebase, without exceeding the frontier-model budget.

Q: Why can routed models outperform one frontier model?

Routed models can outperform a single frontier model when their capabilities complement one another. Different models have different strengths across subdomains because their training corpora, teachers, and post-training tasks differ. A planner can therefore delegate specialized work to models suited to particular subtasks. The panel says this approach can achieve up to 10% higher accuracy on LM Router Bench, depending on the task and model pool.

Q: Why is routing by the initial task type fragile?

Routing by the initial task type is fragile because agentic sessions often change as work progresses. A user may first ask how a codebase works, then request a feature, and finally ask for live testing and deep debugging. A model selected only for the opening question can become poorly suited to later stages, forcing a switch to a stronger model after time and tokens have already been spent.

Q: Why are benchmark scores insufficient for model routing?

Benchmark scores are insufficient because model capabilities are jagged rather than uniformly ordered. One model may score higher on a broad coding benchmark but still be weaker on a particular library, visualization tool, modeling task, or workflow stage. Effective routing therefore requires detailed knowledge of each model's failures, strengths, efficiency, and cost instead of assuming that one benchmark winner is best for every task.

Q: Can a cheaper model cost more than a frontier model?

A cheaper model can cost more overall when it is pushed outside its training distribution. The description says such a model may thrash, repeatedly calling tools in loops until its total cost exceeds what the expensive model would have consumed. It also reports that Opus scored about three times better than Haiku on Terminal Bench at a tenth of the cost, despite Haiku's lower per-token price.

Q: How do cheaper models support deeper codebase exploration?

Cheaper models support deeper exploration because their lower token cost allows the system to spend more tokens within the same budget. Cognition describes using three subagents to explore a codebase, potentially producing broader coverage than a frontier model following one path with limited context. The frontier model can retain responsibility for planning while the implementation models investigate details with greater depth and intensity.

Q: Why does Cognition prefer a continuous sidekick context?

Cognition prefers one sidekick with a continuously running context because it keeps the KV cache warm. According to the description, cached tokens cost roughly ten times less, so preserving context can improve efficiency without repeatedly rebuilding it. This design also avoids relying on separate subagents for every step, allowing the sidekick to retain relevant information as the task develops across multiple stages.

Q: When should an agent compact its context?

An agent should treat compaction primarily as an intelligence decision rather than a simple cost-saving measure. Walden Yan argues that compaction forces a cache miss, giving up the benefits of a warm KV cache. The description also says model quality falls sharply well before the advertised million-token context window, so compaction requires balancing retained context, cache efficiency, and declining performance.

Summary & Key Takeaways

• Model routing addresses the cost of using frontier intelligence by distributing work across multiple models. Cognition keeps a frontier model in charge of planning and difficult decisions while cheaper implementation models perform more extensive work, including parallel codebase exploration. This approach reportedly reduces the cost of Fable-level intelligence by 40%.

• Simple routing based on a task's initial category is fragile because agent sessions evolve. A codebase question can become feature implementation, live testing, and debugging. Models also have jagged capabilities across subdomains, so benchmark leadership does not mean one model is best at every component of a complex workflow.

• Efficiency depends on total system behavior, not merely the advertised price per token. A small model outside its training distribution may repeatedly call tools and ultimately cost more than a frontier model. Continuous context and warm KV caches can lower cached-token costs, while compaction can cause cache misses and reduce model quality.