How to Build AI Agents That Work in Production

TL;DR

Building production-ready AI agents requires broader engineering skills than prompt writing alone. Focus on system design, precise tool contracts, retrieval quality, reliability, security, evaluation, observability, and product thinking, then improve existing agents by tightening schemas and tracing failures back through retrieval, tool selection, and system behavior.

Transcript

I saw a job posting last week that made me laugh. It said, looking for a prompt engineer with experience in distributed systems, API design, machine learning operations, security engineering, and product management. Let's be honest here. That's not a prompt engineering. That's five people. But here's the thing. That job posting isn't wrong. It's ju... Read More

Key Insights

• Agent engineering is the discipline of building complete systems that can make decisions and take real-world actions. Prompt writing is only one component because production agents also depend on tools, databases, models, state management, security controls, and coordinated workflows.
• System design is the foundation that determines how data, models, tools, databases, and sub-agents work together. A sound architecture defines data flow, coordinates specialists, handles component failures, and prevents an agent from becoming an unstructured collection of interconnected parts.
• Tool contract design is essential because vague schemas invite the model to invent missing details. Strict types, required fields, validated patterns, and concrete examples tell the agent exactly what each tool expects and reduce mistakes during sensitive actions such as financial transactions.
• Retrieval quality sets the performance ceiling for agents using Retrieval Augmented Generation. Document chunks that are too large dilute important details, while chunks that are too small lose context. Embeddings must represent similar meanings effectively, and re-ranking should promote the most relevant results.
• Reliability engineering protects agents from failed APIs, unavailable services, network timeouts, and endless retries. Retries with backoff reduce pressure on failing services, timeouts prevent indefinite waiting, fallback paths provide alternatives, and circuit breakers help stop failures from cascading across the system.
• Security and safety require treating every agent as an attack surface. Input validation can catch malicious or malformed requests, output filters can block policy violations, and permission boundaries can restrict dangerous actions, including unnecessary database writes or sending emails without approval.
• Evaluation and observability make agent improvement measurable. Tracing should record decisions, tool calls, parameters, retrieval results, and execution timelines. Evaluation pipelines should use known-good test cases and track success rate, latency, cost per task, and regressions before deployment.
• Product thinking is necessary because users need understandable boundaries, uncertainty signals, graceful errors, clarification, and human escalation. Agent experiences must account for unpredictable performance while setting appropriate expectations and building enough trust for people to use the system in real work.

Questions & Answers

Q: What skills are needed to build production AI agents?

Production AI agents require seven connected skills: system design, tool and contract design, retrieval engineering, reliability engineering, security and safety, evaluation and observability, and product thinking. Together, these skills address architecture, external actions, contextual information, failures, attacks, measurable performance, and the human experience. Prompt engineering remains relevant, but it is only the minimum starting point.

Q: Why is prompt engineering insufficient for production agents?

Prompt engineering focuses on crafting instructions, but production agents do more than answer questions. They book flights, process refunds, query databases, make decisions, and interact with external systems. These actions introduce architectural, reliability, security, measurement, and usability requirements. Better wording cannot compensate for an unclear tool schema, irrelevant retrieval results, failed APIs, excessive permissions, or missing traces.

Q: How does system design improve an AI agent?

System design gives an AI agent a clear structure for coordinating its language model, tools, databases, stored state, additional models, and possible sub-agents. It defines how data moves through the system, how multi-specialist tasks are coordinated, and what happens when a component fails. This prevents the agent from becoming fragile, tangled software that works only during controlled demonstrations.

Q: How should tool contracts be designed for AI agents?

Tool contracts should state exactly what inputs are required and what output will be returned. Schemas should use strict types, required fields, validated patterns, and concrete examples. A user identifier, for example, should not merely be described as a string if only a specific format is valid. Precise contracts reduce ambiguity and prevent the model from filling gaps with invented values.

Q: How can retrieval quality be improved in a RAG agent?

Retrieval quality can be improved by carefully choosing document chunk sizes, verifying that the embedding model places similar concepts near one another, and adding a re-ranking pass. Large chunks can dilute important details, while small chunks can remove necessary context. Re-ranking scores retrieved results for actual relevance and moves stronger evidence above irrelevant material before it reaches the model.

Q: How can AI agents handle API and network failures?

AI agents can handle external failures by using retries with backoff, explicit timeouts, fallback paths, and circuit breakers. Backoff prevents repeated requests from hammering a failing service, while timeouts stop the agent from waiting indefinitely. Fallbacks offer a plan B when the preferred route fails, and circuit breakers help keep one failure from cascading through the entire system.

Q: How should AI agents be protected from prompt injection?

AI agents should use input validation to detect malicious or malformed requests, output filters to block responses that violate policy, and permission boundaries that restrict what the system can attempt. An agent should not automatically receive database write access or permission to send emails. Limiting capabilities reduces damage if malicious instructions override or confuse the model's intended behavior.

Q: How can teams evaluate and debug AI agent performance?

Teams should trace every decision, tool call, parameter, retrieval result, and step in the agent's execution timeline. They should also maintain evaluation pipelines with test cases that have known-good answers and track metrics such as success rate, latency, and cost per task. Automated regression tests can then detect performance declines before changes are deployed to users.

Summary & Key Takeaways

• Production AI agents are systems that make decisions, call tools, store state, query databases, and take real actions. Their architecture must coordinate models, tools, data, and possibly sub-agents while accounting for component failures and multi-step tasks. Prompt engineering remains useful, but it is only the starting point.

• Reliable agent behavior depends on strong infrastructure. Tool contracts need strict inputs, required fields, patterns, and examples. Retrieval systems need effective chunking, meaningful embeddings, and re-ranking. External calls require retries with backoff, timeouts, fallback paths, and circuit breakers so isolated failures do not destabilize the entire system.

• Safe, useful agents require permission boundaries, validation, filtering, tracing, evaluation, and thoughtful user experiences. Teams should measure success rate, latency, and cost per task while recording decisions and tool calls. Agents must also communicate uncertainty, request clarification when needed, escalate appropriately, and handle failures gracefully.