How to Evaluate AI Agents with ADK and Vertex AI

TL;DR

Evaluate an AI agent by measuring task success, output quality, reasoning, tool use, and memory rather than checking only its final response. Combine offline golden-dataset testing with online production monitoring, and balance deterministic checks, model-based judging, and human review. ADK supports a fast local workflow for creating cases, tracing failures, refining instructions, and validating fixes, while Vertex AI supports evaluation at production scale.

Transcript

This week on the agent factory. So LM evaluation is like a school exam. You're testing knowledge with static Q&A. But agent evaluation is more like a job performance review. And that's it. This is uh how you can cover the full loop with dedicated web. As we saw, it's very fast. So this end to end evaluation is one of the most important things in mu... Read More

Key Insights

• Agent evaluation is a system-level assessment of whether an agent completes tasks effectively, makes sound decisions, uses tools and memory correctly, reasons across multiple steps, and responds appropriately to unpredictable situations. A correct-looking final answer alone cannot demonstrate that the underlying behavior is dependable.
• Traditional software testing is designed for deterministic logic, where the same input usually produces the same output. Agents are variable, so strict equality assertions can create flaky test suites. Their performance must instead be examined across behaviors, repeated runs, intermediate decisions, and outcomes over time.
• LLM evaluation measures general model capabilities through static questions and answers, while agent evaluation examines practical job performance. A capable model can still power a poorly performing agent if the surrounding system calls the wrong API, passes incorrect parameters, mishandles context, or fails to recover from errors.
• A full-stack evaluation measures task success, output coherence, factual accuracy, safety, hallucination risk, bias, planning consistency, tool selection, parameter correctness, efficiency, and memory. These dimensions reveal whether an agent completed its goal well or merely reached an acceptable answer through luck.
• Offline evaluation uses static golden datasets before deployment to identify regressions, while online evaluation monitors live user data after deployment to detect drift and support comparative tests. Both are necessary because pre-production cases and real production behavior expose different kinds of weaknesses.
• Ground-truth checks are fast, inexpensive, and reliable for objective requirements such as valid JSON and schema compliance. They are less effective for nuanced qualities such as coherence and factuality, so they should be combined with methods capable of judging subjective or domain-specific characteristics.
• A calibration loop combines human expertise with scalable model-based judging. Domain experts first create a small, high-quality golden dataset, then the model-based judge is adjusted until its scores align with human expectations. This approach combines the accuracy of expert review with scalable automated evaluation.
• ADK Web supports an iterative five-step workflow consisting of defining a golden path, running an evaluation, inspecting the trace, fixing the agent, and validating the change. In the demonstrated case, the trace identified the wrong tool choice, and clearer tool-selection instructions corrected the customer-facing response.

Questions & Answers

Q: What is agent evaluation?

Agent evaluation is the assessment of an agent as a complete system, not merely a check of its final response. It examines whether the agent completes its assigned task, makes good decisions, follows a logical plan, uses tools with correct parameters, retains relevant context, resolves conflicting information, and behaves reliably when conditions are unpredictable.

Q: How does agent evaluation differ from traditional software testing?

Traditional software testing usually checks deterministic behavior, where the same input produces the same expected output and a test clearly passes or fails. Agents are variable and may produce different outcomes from the same prompt. Strict response equality can therefore create flaky tests, so evaluation must examine behavior, decisions, intermediate steps, and performance across repeated runs.

Q: How is agent evaluation different from LLM evaluation?

LLM evaluation tests general model capabilities, often through static questions and answers that resemble a school exam. Agent evaluation resembles a job performance review because it measures whether the whole system can use tools, recover from errors, remain consistent across turns, and complete tasks effectively. A strong model can still support a weak agent implementation.

Q: What metrics should be used to evaluate an AI agent?

An AI agent should be measured across task success, output quality, planning, reasoning, tool utilization, efficiency, memory, and context retention. Evaluation should ask whether the result is coherent, accurate, and safe, whether the plan follows logical steps, whether tools and parameters are correct, whether calls are redundant, and whether conflicting information is resolved appropriately.

Q: What is the difference between offline and online agent evaluation?

Offline evaluation occurs before production and tests the agent against a static golden dataset to identify failures and regressions. Online evaluation happens after deployment and monitors behavior using live user data. It can reveal drift and support comparative testing. Together, these approaches connect controlled pre-production validation with evidence from real-world agent use.

Q: When should ground-truth checks, model judges, and human reviews be used?

Ground-truth checks work well for objective requirements such as valid JSON or schema compliance because they are fast, inexpensive, and reliable. A strong model can judge subjective qualities such as plan coherence at scale, although its scoring depends on its preparation. Human experts provide specialized review and high-quality expectations, but their work is slower and more expensive.

Q: How does a calibration loop improve agent evaluation?

A calibration loop begins with domain experts creating a small, high-quality golden dataset that represents the expected standard. A model-based judge is then adjusted until its scores align with those human expectations. The resulting workflow combines the accuracy and specialization of human review with the scalability of automated judging, compensating for the limitations of either method alone.

Q: How do you evaluate and debug an agent with ADK Web?

Use ADK Web to define a golden path, save a corrected expected response as an evaluation case, run the evaluation, and inspect the trace when the case fails. The trace exposes the agent's step-by-step process and tool choices. After identifying the cause, refine the agent's instructions, rerun the same case, and confirm that the evaluation passes.

Summary & Key Takeaways

• Agent evaluation examines system-level task effectiveness because agents can produce different outcomes from the same prompt. A complete assessment measures whether the agent finishes its task well, develops a logical plan, selects tools and parameters correctly, avoids redundant calls, retains relevant context, resolves conflicting information, and handles unpredictable situations reliably.

• Evaluation should cover both offline and online settings. Offline evaluation uses static golden datasets before production to detect regressions, while online evaluation monitors live user data after deployment for drift and supports comparative testing. Ground-truth checks, model-based judges, and expert human reviews offer different balances of speed, nuance, scalability, accuracy, and cost.

• ADK Web enables a five-step evaluation loop: establish a golden path, evaluate the agent, inspect traces to identify the root cause, fix the agent, and rerun the evaluation. In the product research example, tracing exposed an incorrect tool choice, and clearer instructions separated customer-facing descriptions from internal SKU lookups.