How to Measure AI Agent Reliability and ROI

TL;DR

Build observability into AI agents from the start by combining end-to-end tracing, trace-based evaluation, continuous monitoring, and optimization. Microsoft Foundry applies these capabilities across supported agent frameworks, while Azure Monitor extends visibility across applications, data, infrastructure, and other Azure resources. Context-specific rubric evaluators can assess multiple quality dimensions and help convert production traces into signals for continuous improvement and business-value measurement.

Transcript

o our session from observability to ROI for AI agents on any framework, we're excited to walk you through Microsoft Foundry observability. We'll start with an introduction, but then we'll get right into a variety of demos that cover the end to end agent DevOps life cycle, where we'll walk you through how you can observe any agent on any framework. ... Read More

Key Insights

• Agent observability is necessary because nondeterministic behavior creates reliability and consistency challenges that traditional monitoring cannot fully address. Building observability into production agents makes their execution, response quality, safety, and operational behavior measurable before failures become the only source of diagnostic information.
• Microsoft Foundry Observability is organized around four connected pillars: tracing, evaluation, monitoring, and optimization. Tracing captures execution workflows, evaluation measures quality and safety from those traces, monitoring applies evaluations online to detect issues, and optimization turns the resulting evidence into continuous improvements.
• Tracing is the foundation for understanding an agent's full execution workflow. Foundry uses trace data to expose agent behavior and produce evaluation signals, allowing developers and operators to investigate how a response was generated rather than judging only the final text returned to a user.
• The rubric evaluator creates a context-specific, multidimensional assessment from information such as an agent's system prompt, selected model, target, and optional context files. In the demonstration, the generated rubric contained eight dimensions with different weights and could evaluate multiple aspects through one evaluator.
• Continuous evaluation works by retrieving production traces from Application Insights and running configured evaluators against them. Scores can vary according to the user's question, helping teams identify specific low-performing interactions and inspect their prompts, responses, task-completion results, and rubric explanations.
• Low evaluation scores can reveal unsupported agent claims. In the vendor-analysis demonstration, the agent lacked the exact data requested by the user but still claimed that the vendor was performing well, so trace inspection exposed a response characterized as hallucination rather than a supported conclusion.
• Cross-framework support allows Foundry to observe agents built with several popular frameworks, including LangGraph, the OpenAI SDK, and Microsoft Agent Framework. The stated goal is complete visibility into agent execution workflows and evaluation signals, regardless of the supported framework used to create the agent.
• Full-stack observability connects Foundry with Azure Monitor. Foundry focuses on applications, agents, and AI platform components, while Azure Monitor provides visibility across Azure resources, data, and infrastructure. Foundry can then read that monitoring data back to centralize observability for AI workloads.

Questions & Answers

Q: Why do production AI agents need observability?

Production AI agents need observability because their nondeterministic behavior introduces reliability and consistency challenges for developers and operators. Observability provides evidence about complete execution workflows, response quality, safety, and production health. Without those signals, teams may see a flawed answer without understanding how it arose. Traces, evaluations, continuous monitoring, and optimization create a feedback loop for detecting and correcting such problems.

Q: What are the four pillars of Microsoft Foundry Observability?

The four pillars are tracing, evaluation, monitoring, and optimization. Tracing shows the agent's full end-to-end execution workflow. Evaluation operates on traces to assess agent quality and safety. Monitoring applies the same evaluation methods in an online production setting to detect issues as they occur. Optimization uses those observations and evaluation signals to support continuous improvement of the agent.

Q: How does trace-based evaluation help diagnose an AI agent?

Trace-based evaluation connects quality scores to the execution record that produced an agent response. A developer can select a low-scoring trace, review measures such as task completion and a context-specific rubric score, and inspect the user's actual request and the agent's answer. This makes it possible to determine whether the response used the required data or produced an unsupported conclusion.

Q: What is the Microsoft Foundry rubric evaluator?

The rubric evaluator is a public-preview Foundry capability that automatically generates a multidimensional, context-specific evaluator. The demonstrated setup supplied the agent's system prompt, a selected model, and the hosted agent as its target, with an option to upload additional context files. The resulting rubric contained eight weighted dimensions and could be used for offline testing as well as online evaluation.

Q: How can teams evaluate an agent when they have no existing data?

Teams can begin by generating a rubric evaluator from the context already available for the agent, including its system prompt, model, target, and optional files. This produces evaluation dimensions without requiring an existing collection of production interactions. Once traces become available, the same context-specific evaluator can score them, supporting a path from initial testing to continuous production evaluation.

Q: How did observability reveal hallucination in the vendor analyst agent?

A production trace received low scores for both task completion and the vendor-history rubric. Reviewing the associated interaction showed that the user requested specific information that the agent did not possess. Despite lacking the requested data, the agent stated that the vendor was performing well. The trace and evaluation explanation exposed that favorable conclusion as unsupported, which the presenters characterized as hallucination.

Q: Which agent frameworks can Microsoft Foundry observe?

The session states that Foundry can evaluate its own agents and agents built with supported external frameworks. Examples named in the presentation include LangGraph, the OpenAI SDK, and Microsoft Agent Framework. Its observability experience is intended to provide visibility into complete agent execution workflows and to derive evaluation signals directly from traces across this open framework ecosystem.

Q: How do Microsoft Foundry and Azure Monitor work together?

Microsoft Foundry provides observability for applications, agents, and AI platform components. Its integration with Azure Monitor extends that view across Azure resources, data, infrastructure, and the broader technology stack. Observability data flows to Azure Monitor for full-stack visibility, and Foundry reads relevant data back so teams can use a centralized observability experience for their AI workloads.

Summary & Key Takeaways

• Microsoft Foundry Observability addresses the reliability and consistency challenges created by nondeterministic agents. Its four core pillars are tracing, evaluation, monitoring, and optimization. Traces expose complete execution workflows, evaluations assess quality and safety, monitoring detects production issues, and optimization uses collected signals to improve agent behavior continuously.

• The demonstrated vendor history analyst summarizes job history, surfaces insights, and highlights lessons for vendor managers. A generated rubric evaluates responses across eight weighted dimensions. When one production trace received low task-completion and vendor-history scores, inspection showed that the agent lacked requested data but still presented a favorable assessment, indicating hallucinated content.

• Foundry supports observability for agents built with frameworks including LangGraph, the OpenAI SDK, and Microsoft Agent Framework. Production traces can flow through Application Insights and Azure Monitor, enabling broader visibility across AI workloads and Azure resources. The same evaluators can be used during offline development and continuous online evaluation.