Why Do AI Agents Fail When the Model Is Right?

TL;DR

Reliable AI agents require the harness to own state, serialize mutations, bound every external operation, scope approvals, and preserve receipts. A fluent response or successful internal tool call does not prove that state was persisted or that the user saw the result, so production systems need replayable records and evidence of each terminal outcome.

Transcript

[music] Thank you for choosing to spend this session with me. My goal today is simple. I want to convince you all that most of the production failures are not most of most of the agent failures are not model failures. Those are harness failures. So let's start with one production incident. The user saw the reply. The system forgot it happened. This... Read More

Key Insights

• A model is a probabilistic planner, while the harness is responsible for production control. The model may propose a message, command, tool call, or edit, but the surrounding system must enforce authority, commit state changes in order, and retain evidence of the outcome.
• Silent success is more dangerous than an obvious crash because it gives users and operators no clear failure boundary. A reply can be delivered while persistence fails, leaving the next turn to reconstruct its context from an incomplete record while still producing fluent, confident language.
• State ownership is the ability of a named system of record to reconstruct reality. Storage location alone is insufficient. Every fact that an agent may use later needs one owner and one replay path, or the system cannot reliably claim that it remembered the fact.
• One ordered commit path is required for each mutable state boundary. Two writers can make locally correct changes after reading the same old state, yet the later save can silently erase the earlier change. Serialization can use a queue, mutex, lock, or transaction.
• Concurrency is compatible with reliable agent systems when it is applied outside the critical commit boundary. Parallel reads, independent retrieval, multiple sessions, and sub-agent fan-out can proceed together, while writes to the same mutable state must pass through a single ordered path.
• Silence is not a valid terminal state for an agent run. A missing tool result can leave a session waiting for an event that will never arrive, with later messages queued behind it. Deadlines, cancellation, watchdogs, timeout results, and recovery commands make this failure visible and bounded.
• Approval is scoped execution state, not a vague memory that a person clicked a button. A useful approval identifies the approver, session, run, tool, arguments, validity period, outcome, and related receipt so retries or callbacks cannot separate authority from the authorized action.
• A receipt is stronger evidence than a transcript or an internal success response. The transcript records what the agent said, while the receipt records what policy allowed, what execution attempted, the terminal result, and whether the user-facing surface confirmed that the action actually became visible.

Questions & Answers

Q: Why can an AI agent fail even when its model is correct?

An AI agent can fail because the model is only one part of the production system. The harness assembles context, checks authority, invokes tools, orders state changes, persists results, and delivers output. If any of those responsibilities fails, the model may still answer coherently while relying on stale history, losing a mutation, waiting indefinitely, or claiming an action the user never saw.

Q: What is an agent harness responsible for?

An agent harness is responsible for turning model proposals into controlled production actions. It maps incoming events to sessions, reconstructs the working context, manages tools, applies policies and approvals, orders mutations, persists state, and records audit evidence. The model can propose an action, but the harness must decide whether it is authorized, execute it correctly, and preserve proof of the result.

Q: Why is delivering a reply different from remembering it?

Delivering a reply proves only that output reached a communication path. It does not prove that the conversation turn or associated state was written to the system of record used to build future context. If persistence fails silently, the user sees success while the next turn lacks the relevant fact, allowing the agent to respond confidently from an incomplete history.

Q: How should an AI agent manage shared mutable state?

Shared mutable state should have one ordered commit path for each state boundary. Parallel reads, independent retrieval, multiple sessions, and sub-agent work can still run concurrently, but competing writes to the same state must be serialized. A queue, mutex, lock, or transaction can protect the commit so one locally correct save does not silently erase another locally correct change.

Q: How can an agent recover from a tool call that never returns?

An agent needs deadlines, cancellation, watchdogs, tool timeouts, and explicit error results so a missing response cannot leave the run waiting forever. Recovery commands must also avoid sitting behind the blocked work they are meant to repair. Every external operation should finish as success, failure, timeout, cancellation, or maximum attempts, with that terminal outcome written into the receipt.

Q: What information should an agent approval contain?

An approval should identify who approved the action, the relevant session and run, the specific tool and arguments, how long the authorization remains valid, the resulting outcome, and the associated receipt. Keeping these fields together prevents retries, replays, restarts, or channel callbacks from applying an expired or unrelated approval to an action that was never actually authorized.

Q: What is the difference between a transcript and a receipt?

A transcript records what the agent said, but it does not prove that an authorized mutation occurred or that a message became visible to the user. A receipt records the durable execution chain, including what policy allowed or denied, what the system attempted, how the operation terminated, and what the user-facing edge confirmed. It therefore provides evidence that survives the turn.

Q: How should teams investigate an AI agent production incident?

Teams should ask what event woke the agent, what state it inherited, what authority it used, what action actually executed, and what evidence survived. These questions distinguish model output from harness behavior and reveal failures in context reconstruction, persistence, ordering, lifecycle management, approval scope, or delivery. The investigation should rely on replayable state and receipts rather than conversational confidence alone.

Summary & Key Takeaways

• Agent failures can occur even when the model gives a coherent answer. A response may reach the user while the corresponding state never enters durable history. Because the next turn is assembled from stored transcripts, memory, policy, and tool definitions, missing or stale inputs can produce confident answers based on incomplete reality.

• Reliable state requires a named system of record and a replay path for every fact the agent may need later. Shared mutable state also needs one ordered commit path. Parallel reads and independent work can continue, but competing writes must be serialized with mechanisms such as queues, mutexes, locks, or transactions.

• Production actions require bounded lifecycles, scoped authority, and durable evidence. Every external boundary should end in success, failure, timeout, cancellation, or maximum attempts. Approvals must remain attached to the exact action authorized, while receipts should record what policy allowed, what execution attempted, and what the user-facing surface confirmed.