How Does AgentOps Manage AI Agents at Scale?

TL;DR

AgentOps manages production AI agents through three ordered layers: observability, evaluation, and optimization. Teams first trace actions and costs, then assess completion, accuracy, and safety, and finally improve prompts, retrieval, workflows, and handoffs. In the healthcare example, this approach supports faster prior authorization, measurable quality, lower processing costs, and controlled escalation to human specialists.

Transcript

Your AI agent just approved a prescription. Or did it deny one? Actually, do you even know? Because right now most teams running agents in production are flying blind. And in healthcare, finance, anywhere with real stakes, blind is not a strategy—it's a liability. Let me paint you a picture. A patient needs a specialty medication. Their doctor pres... Read More

Key Insights

• AgentOps is the discipline of managing, monitoring, evaluating, and improving AI agents after deployment. It is intended for systems that take real-world actions, including opening tickets, updating records, making decisions, and calling APIs, where teams must understand what happened and whether it should have happened.
• The AgentOps framework has three ordered layers: observability, evaluation, and optimization. The order matters because teams cannot improve behavior they cannot measure, and they cannot measure behavior they cannot see through detailed traces of agent activity.
• Observability is the visibility layer for reconstructing an agent's decisions. It records tool calls, LLM invocations, and handoffs, while tracking end-to-end trace duration, agent-to-agent handoff latency, tool execution latency, and the API cost associated with each request.
• Evaluation is the quality layer that determines whether an agent performed its task correctly and safely. Its central measures include task completion rate, guardrail violation rate, factual accuracy, human-reviewed clinical appropriateness, and the proportion of submissions approved on the first attempt.
• Optimization is the improvement layer applied after behavior becomes visible and measurable. It examines prompt token efficiency, retrieval precision at K, workflow step efficiency, handoff success, and improvement velocity to reduce cost and friction without sacrificing output quality.
• The healthcare workflow uses two cooperating agents to process prior authorizations. A clinical documentation agent gathers diagnosis codes, laboratory results, and failed treatments, while a payer authorization agent submits the package, monitors insurer responses, requests additional information, and notifies the pharmacy and doctor.
• The prior authorization example achieves an average completion time of 2.8 hours, representing an 85% reduction from the manual three-to-five-business-day process. It also completes 94.2% of requests without human intervention and routes the remaining 5.8% to specialists.
• The example's optimization data identifies specific improvement opportunities. Prompt length falls from 1,800 to 1,100 tokens with the same quality score, retrieval precision at five reaches 0.84, and a 98.7% handoff success rate reveals that EHR unavailability causes most failures.

Questions & Answers

Q: What is AgentOps for AI agents?

AgentOps is the emerging discipline of managing AI agents in production, not merely deploying them. It provides practices for monitoring actions, evaluating results, improving performance, and detecting failures before users encounter them. It is especially relevant when agents call APIs, update records, open tickets, make decisions, or interact with external systems and other agents.

Q: What are the three layers of AgentOps?

The three AgentOps layers are observability, evaluation, and optimization. Observability makes agent behavior visible and reconstructable. Evaluation determines whether the observed behavior was accurate, safe, and successful. Optimization uses those measurements to improve prompts, retrieval, workflows, costs, and handoffs. The sequence matters because improvement depends on measurement, and measurement depends on visibility.

Q: What metrics should AgentOps observability track?

AgentOps observability should track end-to-end trace duration, agent-to-agent handoff latency, tool execution latency, and cost per request. Detailed traces should also capture every tool call, LLM invocation, and handoff. Together, these records show how long work takes, where bottlenecks appear, how external systems perform, and how much API usage each interaction consumes.

Q: How does AgentOps evaluate whether an AI agent is reliable?

AgentOps evaluates reliability through task completion rate, guardrail violation rate, and factual accuracy, supported by domain-specific review where needed. In the healthcare example, pharmacists review 5% of submissions and rate 97.3% as clinically appropriate. Diagnosis codes reach 99.4% accuracy, lab values reach 99.8%, and source records provide the validation baseline.

Q: How do AI agents automate healthcare prior authorization?

A clinical documentation agent connects to the hospital EHR and assembles diagnosis codes, laboratory results, and evidence of previous treatments that failed. A payer authorization agent submits that package to the insurance portal, monitors its status, and requests additional documentation when necessary. After a decision, it notifies the pharmacy and doctor, while unusual cases escalate to specialists.

Q: What results did the AgentOps prior authorization example achieve?

The example reports an average authorization time of 2.8 hours, down 85% from a manual process lasting three to five business days. It completes 94.2% of requests without human involvement, with 5.8% escalating to specialists. Each authorization costs 47 cents in API usage, compared with 25 dollars for manual processing.

Q: How does AgentOps reduce AI agent costs?

AgentOps reduces costs by exposing token usage, unnecessary workflow steps, irrelevant retrieval results, and failed handoffs. In the example, prompt tuning reduces prompt length from 1,800 tokens to 1,100 while maintaining the same quality score, producing a 39% cost reduction per request. The dashboard also measures a total API cost of 47 cents per authorization.

Q: How does AgentOps identify workflow optimization opportunities?

AgentOps compares actual workflows with efficient target paths and measures retrieval and handoff performance. The prior authorization workflow averages 7.2 steps against an optimal six because incomplete initial EHR queries trigger follow-ups. Retrieval precision at five is 0.84, and handoff success is 98.7%, with most failures linked to EHR unavailability and a need for better retry logic.

Summary & Key Takeaways

• AgentOps provides the operational infrastructure needed to manage AI agents that call APIs, update records, make decisions, and interact with other agents. Its three layers must be implemented in order: observability reveals what happened, evaluation determines whether the result was acceptable, and optimization improves performance using evidence from the first two layers.

• Observability reconstructs every tool call, model invocation, and agent handoff. Evaluation measures task completion, factual accuracy, guardrail violations, clinical appropriateness, and first-pass approval. Optimization targets token efficiency, workflow steps, retrieval precision, handoff reliability, and improvement velocity, allowing teams to make agents faster, cheaper, safer, and more accurate over time.

• In the prior authorization example, two agents gather clinical evidence and communicate with an insurer. Average processing reaches 2.8 hours, 94.2% of requests finish without human intervention, and each authorization costs 47 cents in API usage. Human specialists handle exceptions while pharmacists independently review a sample for clinical appropriateness.