How to Build a Voice AI Agent to Negotiate Deals with VAPI

TL;DR

A single voice AI agent named Alex placed over 800 calls in one day to secondhand luxury watch dealers, lowballing them on a $35,000 Rolex Daytona and logging every offer to a database. It was built on VAPI using Gemini 2.0 Flash and a Cartesia voice, with tool calls automatically flagging which calls produced real offers.

Transcript

I brought in my main automation man's Tony who works with SA Checkout. Tony, what are we going to learn today? Uh, I'm going to tell you the story of the time that I made a voice AI that lowballed every single luxury watch dealer in the US for a Rolex Daytona. That's a $35,000 watch. And uh, it actually kind of worked. Are you the seller with the R... Read More

Key Insights

• Voice AI agents can automate phone-based negotiation at scale: one agent named Alex placed over 800 calls in a single day to secondhand luxury watch dealers, attempting to lowball them on a Rolex Daytona valued at $35,000.
• VAPI (Voice API) is a flexible platform for building voice agents that avoids proprietary lock-in. You bring your own prompt, pick providers for each service (model, voice), and hit go to run the agent.
• Prompt engineering for voice AI is iterative and reactive. Tony refined his prompt by simulating lowball conversations, spotting bad responses, and patching them one by one, comparing it to plugging holes in a sinking boat.
• Instructing the model to use sixth grade English produced noticeably longer call durations and more human-sounding conversations, because most human phone interactions happen at roughly a sixth grade level.
• Keeping responses concise, one to two sentences maximum, stops the agent from ramming every question at once, which sounds obviously robotic and reduces the chance the voice provider mangles the output.
• Temperature controls response creativity: near zero makes the system deterministic and predictable but poor at handling uncommon questions, while higher values let it respond appropriately to unexpected inputs. Tony ran his at temperature one.
• Tool calling is the most powerful part of VAPI because it lets the voice AI interact with existing tools. Tony used a tool to analyze each transcript and log whether a call produced an offer to an Airtable database.
• Model choice is often flavor-of-the-month and depends on availability. Tony used DeepSeek before its hype cycle to save money, but heavy API load later caused his bot to crash and stop responding to calls.

Questions & Answers

Q: What is Lowballer 9000 and what did it do?

Lowballer 9000, nicknamed Alex, is a voice AI agent built by Tony Ge to call secondhand luxury watch dealers across the US and negotiate down the price of a Rolex Daytona, a watch valued at $35,000. In a single day the agent placed over 800 calls, gathering critical details such as the watch's condition and whether it came with original documentation and packaging, then attempting to lowball each dealer while logging the results.

Q: What platform was used to build the voice AI agent?

The agent was built on VAPI, which Tony explains basically amounts to 'voice API.' It is described as a very flexible way to build voice agents that does not lock you into anything proprietary. You simply bring your own prompt, pick your providers for the different services such as the language model and the voice, and then hit go to run it. VAPI's own documentation also offered general best practices for building prompts.

Q: Why does using sixth grade English improve voice AI calls?

Tony found that most human phone interactions happen at roughly a sixth grade English level. When he changed the prompt to instruct the model to only use sixth grade English, he got significantly longer call durations and the agent sounded much more human when he talked to it. Although he did not have precise metrics, the change made conversations feel far more natural and engaging rather than obviously robotic.

Q: How should you choose the model and temperature for a voice agent?

Tony says model choice is very dependent and often flavor-of-the-month, so you should also pick one that is reliably available. He once used DeepSeek to save money, but heavy API load during its hype cycle made his bot crash and stop responding. For temperature, which controls how creative responses are, a value near zero makes the system deterministic and predictable but poor at handling uncommon questions, so he ran his at temperature one using Gemini 2.0 Flash.

Q: What are tool calls and why are they important in VAPI?

Tony calls tool calling the most powerful part of VAPI, because a voice AI becomes genuinely useful when it can interact with your existing tools. He created a tool that, once a conversation ends, takes the entire transcript and determines whether the call produced an offer, whether the dealer engaged, whether they were lowballed and by how much, and whether they accepted or gave a counter price. These results are recorded automatically so useful calls surface without manual sifting.

Q: How did Tony manage the data from over 800 calls?

Tony built a database he calls his watch list, using Airtable, that records every single call the AI placed. Rather than manually sifting through more than 800 calls, a tool call analyzes each transcript and populates a spreadsheet showing which calls resulted in an actual offer, how much the dealer was lowballed, and whether they accepted or gave their lowest price. A 'get best offer' tool can then feed that data back into ongoing negotiations.

Q: How does Tony think voice AI will affect phone-based work?

Tony believes people are drastically underestimating what voice AIs are capable of. His stated belief is that any work you do over the phone could and may get automated within the next one to three years. His Rolex Daytona experiment, where a single agent negotiated with hundreds of dealers in a day, is meant to demonstrate that voice AI can already handle real, high-stakes phone negotiations at scale.

Q: What voice did Tony use for the agent and why?

Under VAPI's voice configuration, Tony used Cartesia, selecting a voice they call 'New York man.' He jokingly references it sounding like a Sopranos character, noting the voice has an Italian, New York or New Jersey quality. VAPI offers all the popular voice providers, but choosing a distinctive, human-sounding voice helped the agent feel more believable to the luxury watch dealers it was calling during negotiations.

Summary & Key Takeaways

• Tony Ge built a voice AI agent called Lowballer 9000, named Alex, to negotiate down secondhand luxury watch dealers across the US on a Rolex Daytona worth $35,000. The agent gathers critical details like the watch's condition and whether it has original documentation and packaging, since these dramatically affect the quoted price.

• The system runs on VAPI, a flexible voice API platform where you supply your own prompt and pick providers for the model and voice. Tony used Gemini 2.0 Flash at temperature one and a Cartesia voice styled as a New York man. His prompt was hand-written and continuously refined by simulating real lowball conversations.

• Tool calls make the agent genuinely useful: after each of the 800-plus calls, a tool analyzes the transcript to determine whether the dealer engaged, whether they were lowballed, by how much, and whether they gave a counter price. Results are logged to an Airtable watch list so actionable offers surface automatically.