How to Build Voice Agents with OpenAI Audio Models

TL;DR

Developers can turn existing text agents into voice agents by adding speech-to-text before the agent and text-to-speech after it. OpenAI introduced GPT-4o Transcribe, GPT-4o Mini Transcribe, GPT-4o Mini TTS, streaming transcription, noise cancellation, semantic voice activity detection, and an Agents SDK update, while the commentary notes that direct speech-to-speech systems can better preserve tone and emotion.

Transcript

open aai just announced a bunch of Audio model updates let's watch their live stream and I'll give you my thoughts today is really really exciting we're moving beyond text to voice agents so in a way voice is a very natural human interface and today we're going to enable developers and businesses to build voice agents okay so I completely agree voi... Read More

Key Insights

• Voice is a natural interface for AI systems because users can request information without typing or looking at a screen. The commentary highlights driving and quick, hands-free questions as situations where voice access can be faster and more convenient than a screen-based interaction.
• A voice agent is an AI system that can act independently for a user or developer through spoken conversation. Examples in the transcript include customer support for product catalogs and orders, pronunciation coaching, lesson planning, and simulated conversations for language learners.
• Direct speech-to-speech processing accepts spoken input and produces spoken output without an intermediate transcription step. According to the commentary, this approach can preserve intonation, mood, word emphasis, and emotion, then reflect similar vocal qualities in the generated response.
• The chained voice architecture converts speech into text, sends that text through an agent, and converts the result back into speech. Its advantages are modularity, compatibility with existing text agents, flexible model selection, easier setup, and access to text models described as the intelligence benchmark.
• Text transcription can discard meaningful vocal information such as tone, emotion, and emphasis. The commentary observes that common text conventions, including italics, bold formatting, and emojis, can convey emphasis online, but typical speech-to-text systems do not necessarily use such decorators to preserve vocal nuance.
• GPT-4o Transcribe and GPT-4o Mini Transcribe are new speech-to-text models trained from a large speech model using trillions of audio tokens. The smaller Mini model is described as faster and more efficient while retaining as much of the larger model's transcription capability as possible.
• Word error rate measures the percentage of words a transcription model gets wrong, so a lower rate indicates better accuracy. OpenAI reports that its new transcription generation outperformed Whisper 2 and Whisper 3 across every tested language, with especially large improvements for certain languages.
• Streaming transcription sends a continuous audio stream to the model and receives continuous text in response. The APIs also provide noise cancellation and semantic voice activity detection, reducing interference from background sounds and avoiding processing a user's unfinished thought as a complete request.

Questions & Answers

Q: How can developers turn a text agent into a voice agent?

Developers can place a speech-to-text model before an existing text agent and a text-to-speech model after it. Spoken input is transcribed, the established text agent processes the request, and its response is converted into audio. This chained design reuses prior work, supports mixing and matching components, and is presented as the easiest way to begin building a voice experience.

Q: What is the difference between speech-to-speech and chained voice systems?

A speech-to-speech system takes spoken input and directly produces spoken output through one model. A chained system first converts speech into text, processes that text with an agent, and then synthesizes speech. The direct method can preserve tone and emotion more effectively, while the chain provides modularity, easier setup, flexible model selection, and compatibility with existing text-based agents.

Q: Why can speech-to-text conversion lose important information?

Speech contains more than words, including intonation, emotion, pacing, mood, and emphasis on particular phrases. Converting an utterance into ordinary text can remove those signals before the agent processes the request. The commentary notes that written communication uses devices such as bold text, italics, and emojis for emphasis, but typical transcription does not necessarily preserve vocal nuance through equivalent formatting.

Q: What speech-to-text models did OpenAI announce?

OpenAI announced GPT-4o Transcribe and GPT-4o Mini Transcribe. The larger model is positioned as the more capable transcription option, while the Mini version is distilled to a smaller size for greater speed and efficiency. OpenAI says the new generation uses a large speech model trained on trillions of audio tokens and outperforms earlier Whisper models across tested languages.

Q: How is speech transcription accuracy measured?

Transcription accuracy is evaluated using word error rate, which is the percentage of words the model gets wrong. Lower word error rates represent better performance. In the presentation, OpenAI compared the new GPT-4o transcription models with Whisper 2 and Whisper 3 and reported improvements across every language tested, with particularly substantial gains visible for some languages.

Q: How much do the new OpenAI transcription models cost?

GPT-4o Transcribe is stated to cost 0.6 cents per minute, the same price identified for Whisper, while GPT-4o Mini Transcribe is stated to cost 0.3 cents per minute. The commentary considers these prices good but argues that open-source transcription models may cost less when suitable hardware is available, especially for local processing or production workloads where small accuracy differences are acceptable.

Q: What features make the speech-to-text APIs useful for live voice agents?

The APIs accept a continuous stream of audio and return a continuous stream of text, supporting faster conversational experiences. They also include noise cancellation to reduce problems caused by background sounds. A semantic voice activity detector divides the audio according to when the model believes the user has finished speaking, helping developers avoid processing incomplete thoughts as finished requests.

Q: When might a local open-source audio model be preferable?

A local open-source model may be preferable when an application must process audio directly on the user's computer, avoid hosted per-minute charges, or operate with available local hardware. The commentary says open-source speech models are already effective and may cost only a fraction of hosted services at scale, although OpenAI's managed APIs bundle useful capabilities such as noise cancellation and automatic semantic audio chunking.

Summary & Key Takeaways

• OpenAI announced three audio models and new tools for building humanlike voice experiences. The releases include two speech-to-text models, GPT-4o Transcribe and GPT-4o Mini Transcribe, plus GPT-4o Mini TTS. An Agents SDK update is intended to simplify converting existing text-based agents into voice agents for practical applications.

• Voice agents can use direct speech-to-speech processing or a modular chain that converts speech to text, runs a text agent, and converts its response back to speech. Direct processing better retains vocal information, while chained systems offer modular components, access to strong text models, easier development, and higher reliability.

• The transcription models reportedly outperform Whisper across every tested language and support continuous audio streaming. Their APIs also include noise cancellation and semantic voice activity detection, which determines when a user has finished speaking. The commentary considers these managed features valuable while noting that local open-source models may cost less for some workloads.