What Are AI Evals and Why Do They Matter?

TL;DR

Evals are structured tests that measure how well an AI system performs on quality, reliability, and correctness. They answer whether changing a model or prompt makes an application better or worse, and require three ingredients: a task, a dataset, and scores. Running them offline and online creates a flywheel that cuts dev time and improves quality.

Transcript

Hey everybody, my name is Doug Guthrie. Uh, I'm a solutions engineer at Brain Trust. Um, as you can see here, we're an endto-end developer platform for building AI products. We do evals. If you watched the the keynote this morning, you saw our founding engineer jumping up and down on stage yelling evals. I am not going to do that. I'm not as uh as ... Read More

Key Insights

• An eval is a structured test that checks how well an AI system performs, helping teams measure quality, reliability, and correctness for non-deterministic large language model outputs that are otherwise hard to build with rigorously.
• Every eval requires three ingredients: a task (the code or prompt to evaluate, needing only an input and output), a dataset of real-world examples to run against, and scores that hold the evaluation logic.
• A task in Braintrust can range from a single prompt to a full agentic workflow calling out to tools, with no limit on complexity as long as it produces an input and an output.
• Scores come in two types: LLM-as-a-judge, which rates output against criteria like excellent, fair, or poor mapping to values such as 0, 0.5, or 1, and code-based scores that are more heuristic or binary.
• Offline evals happen pre-production, where teams iterate, define tasks and scores, and identify and resolve issues before deployment to build a baseline they can improve upon.
• Online evals provide real-time tracing in production, logging model inputs, outputs, intermediate steps, and tool calls to diagnose performance, reliability, latency, cost, tokens, and duration.
• The advice for getting started is to skip building a perfect golden dataset and instead create a baseline immediately, then iterate and improve from that established foundation.
• Combining offline and online evals with user feedback creates a flywheel effect where production logs feed back into datasets, cutting development time and enhancing application quality.
• Evals can be viewed as more than unit tests: rather than only playing defense, they let teams play offense by adding rigor to how they build and develop applications worth putting into production.

Questions & Answers

Q: What is an AI eval?

According to Braintrust's definition, an eval is a structured test that checks how well your AI system performs. It helps you measure the things that matter, such as quality, reliability, and correctness. Evals address the challenge that large language models produce non-deterministic outputs, which makes it really hard to build a good application you can confidently put into production without a rigorous evaluation process in place.

Q: What are the three ingredients needed to create an eval?

An eval requires three ingredients. First, a task, which is the code or prompt you want to evaluate and only needs an input and an output. Second, a dataset, which holds your real-world examples that you run the task against to understand performance. Third, scores, which contain the logic behind your eval. Together these components let you measure how well your application performs.

Q: What is the difference between offline and online evals?

Offline evals are pre-production. This is where you do the iteration, define your tasks and scores, and identify and resolve issues before deployment. Online evals are real-time tracing of the application in production. They log model inputs and outputs, intermediate steps, and tool calls, allowing you to diagnose performance, reliability, and latency issues while pulling back metrics on cost, tokens, and duration.

Q: How do scores work in an eval?

Scores hold the logic behind your eval and come in two main types. LLM-as-a-judge scores give the model the output plus criteria, letting it assess whether the output is excellent, fair, or poor, which then correspond to numeric values like 0, 0.5, or 1. Code-based scores are more heuristic or binary. You can use both types together to aid development and ensure you are building a really good application.

Q: Why should you use evals when building with large language models?

Evals help answer key questions: when you change the underlying model or the prompt, is your application getting better or worse? They let you detect regressions in code and move away from building without a rigorous process. Because large language models produce non-deterministic outputs, evals add rigor and let teams play offense rather than just defense, ensuring they build applications good enough for production.

Q: How should someone get started with evals?

Guthrie shares advice he heard from Braintrust's CEO Anker: just get started and create a baseline you can iterate and build from. Many people get caught trying to create a perfect golden dataset of test cases before iterating, but you do not have to do that. Instead, start, build that baseline, and establish a foundation you can then improve upon over time as you develop your application.

Q: What is the flywheel effect in evals?

The flywheel effect comes from running evals both offline and online and connecting them with user feedback. By linking what happens in real life with your users in production, filtering those logs down, and adding spans to datasets, you inform your offline work. This feedback loop cuts development time and enhances the quality of the application you put into production, becoming really powerful from a development perspective.

Q: What can a task be in Braintrust?

A task is the code or prompt you want to evaluate. In Braintrust, a task can be as simple as a single prompt or as complex as a full agentic workflow that calls out to tools. There is no limit on the complexity you put into a task. The only requirement is that it has an input and an output. Within the platform you specify an underlying model, give it a system prompt, and can provide access to tools and mustache templating for variables.

Summary & Key Takeaways

• Doug Guthrie, a solutions engineer at Braintrust, introduces evals as an end-to-end developer platform capability for building AI products. The talk covers why to use evals, their components, running them via the SDK and platform, moving to production with human review and online scoring, and human-in-the-loop user feedback.

• Evals help answer whether changing a model or prompt makes an application better or worse and help detect regressions. They add rigor to building with large language models, whose non-deterministic outputs make production-quality applications hard to achieve without a structured evaluation process.

• An eval needs a task, a dataset, and scores. Offline evals run pre-production for iteration; online evals trace production behavior. Connecting both with user feedback creates a flywheel that filters logs into datasets, cutting dev time and increasing quality. Guthrie advises starting with a baseline rather than a perfect golden dataset.