How Braintrust Evaluates Production AI Systems

TL;DR

Reliable AI products require systematic evaluations built from real user data, not a handful of successful prototype examples. Braintrust gives teams a standardized way to compare prompts and models, inspect complex outputs, monitor production behavior, and improve quality, while production customers commonly use retrieval-augmented generation and generally favor instruction-tuned proprietary models over fine-tuning or open-source alternatives.

Transcript

[Applause] so today on no priors um we have aner Goyle the co-founder and CEO of Brain Trust Anker was previously vice president of engineering at single store and was the founder and CEO of impira an AI company acquired by figma Brain Trust is an endtoend Enterprise platform for building AI applications they help companies like notion air table in... Read More

Key Insights

• Braintrust emerged from a repeated infrastructure need at Impira and Figma, where teams built similar internal systems to evaluate AI behavior, collect real user data, and improve outputs. Encountering the same problem before and after the rise of large language models suggested that evaluation would remain important.
• AI evaluation is more difficult than running examples through a simple loop. As applications incorporate agents and more complicated behaviors, teams must execute evaluations quickly, interpret increasingly complex results, and connect those findings to product changes so they can maintain a fast development cycle.
• The gap between an impressive prototype and a reliable product is uncertainty about quality. A feature may work on several selected examples yet fail after release, so systematic evaluations are needed to reveal weaknesses and guide repeated improvements toward consistently strong production outputs.
• Standardized AI tooling supports adoption across an entire organization. Early customers expected AI to become pervasive rather than remain a single supervised project, so they wanted shared methods that could teach new engineers how to develop and evaluate AI applications consistently.
• Retrieval-augmented generation appears in roughly 50% of the production use cases Braintrust observes. Goyal describes its adoption as clear and widespread among customers that have shipped AI products, distinguishing demonstrated production use from technologies that mainly attract discussion or experimentation.
• Fine-tuning is a technique rather than the desired outcome. Customers generally seek automatic workload optimization, but modifying model weights is slower, more expensive, and capable of damaging performance, while instruction tuning can guide behavior by adding examples and directions to a prompt.
• Instruction-tuned models have largely replaced fine-tuned models among Braintrust customers. The company repeatedly benchmarked fine-tuning on customer workloads, but nearly all customers ultimately moved to instruction-tuned alternatives after access, cost, and performance conditions made those models more practical.
• Open-source model interest is stronger than its practical production adoption. Goyal says customers still make limited use of open-source models, although interest is rising, because production teams primarily optimize for user experience, output quality, iteration speed, and return on model spending.

Questions & Answers

Q: What problem does Braintrust solve for AI teams?

Braintrust helps teams turn uncertain AI prototypes into production systems whose quality can be evaluated and improved systematically. It supports evaluations, observability, prompt development, inspection of complex results, and the use of real user data. The platform also gives organizations a consistent development process, which matters when AI expands beyond one project and becomes a capability used by many engineering teams.

Q: Why are AI evaluations harder than a simple test loop?

AI evaluations become difficult because running examples is only the beginning. Teams must execute tests quickly, inspect outputs that grow more complicated when agents are involved, compare changes, and understand which results improved or regressed. Since faster evaluation and faster interpretation directly support faster iteration, a basic loop with logged outputs does not provide the complete workflow required for dependable production development.

Q: How does Braintrust help move an AI prototype into production?

Braintrust reduces uncertainty by letting teams define evaluations and repeatedly use their results to improve an application. A prototype may look successful on a few examples but perform poorly with real users. Systematic evaluation exposes that gap, provides evidence about output quality, and creates a repeatable improvement cycle that can move a feature from occasional success toward consistently strong behavior.

Q: How common is retrieval-augmented generation in production AI?

Retrieval-augmented generation is clearly established among the production systems Braintrust observes. Goyal estimates that roughly 50% of customer use cases running in production involve some form of retrieval-augmented generation. That estimate reflects enterprises that have actually shipped AI products, so it offers a view of operational adoption rather than only the technologies developers are discussing, testing, or exploring.

Q: What is the difference between instruction tuning and fine-tuning?

Instruction tuning changes a prompt by adding directions and examples that demonstrate the desired behavior. Fine-tuning works at a lower level by modifying or supplementing a model's weights using training examples. Both approaches use data to influence behavior, but fine-tuning is slower, more expensive, and easier to perform incorrectly, potentially making the model worse on real-world use cases.

Q: Why have Braintrust customers moved away from fine-tuned models?

Most customers found instruction-tuned models more practical for achieving strong workload performance. Braintrust re-benchmarked fine-tuning with customers every two to three months, and there was a period when GPT-3.5 fine-tuning offered a useful quality lever because GPT-4 access was difficult. As access and economics changed, nearly all customers moved to instruction-tuned models that delivered good performance with less complexity.

Q: Are enterprises widely adopting open-source AI models?

Practical adoption of open-source models remains limited among the production customers Braintrust observes, although interest is higher than before. Goyal believes the market is approaching an important transition, but says it has not arrived yet. Production teams continue to emphasize the best user experience and fastest iteration speed, and proprietary model fees can be negligible or justified by high returns.

Q: Why do enterprises want standardized AI development tooling?

Enterprises expect AI to spread throughout their organizations rather than remain a feature managed by one specialist or one team. A standardized platform gives engineers a consistent way to evaluate outputs, review changes, and learn sound development practices. Early Braintrust customers valued this consistency because it could help new engineers build AI applications correctly while supporting long-term organizational adoption and vendor reliability.

Summary & Key Takeaways

• Ankur Goyal founded Braintrust after encountering the same AI development problems at Impira and Figma. Both organizations needed internal tools for evaluating outputs, collecting real user data, and improving product quality. Repeated demand from early customers showed that evaluation infrastructure was a persistent requirement across both pre-LLM and post-LLM systems.

• Braintrust addresses the uncertainty between a promising AI prototype and a dependable production feature. Teams can implement evaluations, inspect increasingly complex agent outputs, compare changes, and systematically improve results. Standardized tooling also allows AI development practices to spread across an organization instead of remaining confined to one project or specialist team.

• Production adoption differs from broader developer discussion. Roughly 50% of observed production use cases involve some form of retrieval-augmented generation, while nearly all customers have moved from fine-tuned models toward instruction-tuned models. Proprietary models remain dominant because customers prioritize output quality, user experience, iteration speed, and favorable returns over avoiding per-token fees.