How to Build a Voice Agent With Claude Code

TL;DR

Build an ElevenLabs voice agent by describing its purpose to Claude Code, letting it plan the implementation, configure the persona, voice, knowledge, and tools, then embed the provided widget snippet on a website. The demonstrated agent was assembled in about 15 minutes and could answer questions using 400 YouTube transcripts, while another workflow captured leads and booked discovery calls through cal.com.

Transcript

It has never been so easy to build whatever you want. And so in today's video, I'm going to show you guys how easy it is to build a voice agent with just your natural language using Cloud Code with 11 Labs. So, let me show you guys a quick example. I had this crazy idea where I was thinking, okay, I've got a ton of YouTube videos, exactly 400 actua... Read More

Key Insights

• A voice agent can be created from a high-level natural-language description because Claude Code can research the ElevenLabs documentation, determine an implementation approach, configure the agent, and build the supporting tools without requiring every setting to be entered manually.
• The demonstrated knowledge assistant was built in about 15 minutes and used exactly 400 YouTube transcripts. It could accept spoken questions, search the creator's accumulated content, and return relevant guidance about subjects such as Firecrawl, scraping, and beginner automation workflows.
• A voice agent operates as a repeating conversational loop. It listens through a microphone, transcribes speech into text, lets an LLM interpret the message, optionally reads a database or invokes a tool, delivers an answer, and then waits for the next user message.
• The four main components of a voice agent are its persona, voice, knowledge, and tools. The persona controls behavior, the voice controls how responses sound, the knowledge supplies relevant information, and tools let the agent perform operations beyond ordinary conversation.
• A persona is primarily established through the system prompt. It can identify the assistant, describe its setting and available information, and specify a communication style, including whether the agent should sound conversational, imitate the creator's manner, act rudely, or add jokes.
• Knowledge can come from documents or external data systems. The tutorial discusses YouTube transcripts, customer and order data, Supabase or Pinecone vector stores, and NotebookLM as possible sources that an agent could consult before responding to a caller.
• Tools allow an ElevenLabs agent to take actions or retrieve information. Supported examples in the walkthrough include listing GitHub repositories, connecting to an MCP server, making a direct API call, and invoking an existing n8n workflow, Python script, or Zapier automation.
• Voice agents can be deployed through three main interfaces: direct testing inside the ElevenLabs dashboard, a conversation widget embedded on a website, or a phone connection using Twilio. The phone option can support both calls placed by the agent and incoming calls answered by it.
• More videos from this channel:
  • How Did an AI Creator Reach 1M Subscribers?
  • How to Build a Live RAG Pipeline with Drive and Supabase
  • How to Build a No Code AI Agent with n8n Email
• More from this channel: Nate Herk | AI Automation

Questions & Answers

Q: How do you build an ElevenLabs voice agent with Claude Code?

Open a project in VS Code, install and sign in to the Claude Code extension, and use plan mode to describe the desired outcome in natural language. Claude Code can ask questions, study the ElevenLabs documentation, select an implementation method, and configure the agent. The workflow can cover the persona, voice, knowledge, tools, website integration, API keys, testing, iteration, and debugging.

Q: What are the four main components of a voice agent?

The four main components are persona, voice, knowledge, and tools. The persona is defined mainly through the system prompt and controls the agent's behavior. The voice determines how its speech sounds. Knowledge provides business, customer, or content information. Tools let the agent call external systems, retrieve records, or trigger existing automations when a conversation requires action.

Q: How does an AI voice agent process a conversation?

A voice agent uses a repeating loop rather than a single response process. It listens to a visitor through the microphone, converts the spoken message into text, and lets an LLM interpret it. The agent can then answer directly, consult a database, or call a configured tool. It speaks the resulting answer to the visitor and repeats the process for the next message.

Q: How can a voice agent use YouTube transcripts as knowledge?

YouTube transcripts can be collected and connected to the agent as a searchable knowledge source. In the demonstration, Claude Code pulled exactly 400 transcripts and connected their contents to an ElevenLabs agent. The resulting assistant could answer spoken questions about topics discussed across the channel, including scraping tools, Firecrawl, Claude Code workflows, and practical automation ideas mentioned in the videos.

Q: How can an ElevenLabs voice agent be added to a website?

ElevenLabs provides a widget snippet that can be added to the pages where the conversation interface should appear. The snippet creates a control that lets a visitor start a call with the agent. In the demonstrated workflow, the creator could give that snippet to Claude Code and ask it to place the widget into an existing landing-page project without manually wiring the interface.

Q: What tools can an ElevenLabs voice agent call?

An ElevenLabs voice agent can be configured to call tools that retrieve information or perform actions. Examples shown or discussed include a tool for listing GitHub repositories, an MCP server, a direct API call, an n8n workflow, a Python script, and a Zapier automation. Claude Code can read the relevant documentation and handle much of the tool configuration from a natural-language request.

Q: What are the main ways to deploy a voice agent?

A voice agent can be used in three main ways described in the tutorial. It can be tested directly inside the ElevenLabs dashboard, embedded into a website through a conversation widget, or connected to a phone number through Twilio. With the phone configuration, the agent can make outbound calls from the number or answer when someone calls that number.

Q: Why use Claude Code instead of configuring a voice agent manually?

Claude Code reduces the need to click through dashboard settings and manually configure prompts, messages, knowledge sources, endpoints, and tools. A user can describe the intended result, and Claude Code can research the ElevenLabs documentation and implement an appropriate setup. The demonstrated build also included testing and iteration, while the broader tutorial covered API-key setup and debugging a time-zone problem.

Summary & Key Takeaways

• Claude Code can turn a high-level natural-language idea into a working ElevenLabs voice agent. The demonstrated assistant incorporated 400 YouTube transcripts, answered spoken questions about their content, and appeared on a website through an embedded conversation widget. The initial version was built in about 15 minutes.

• Every voice agent combines four main elements: a persona defined by its system prompt, a selected or cloned voice, relevant knowledge, and tools that perform external actions. During a conversation, microphone input is transcribed, interpreted by an LLM, and answered directly or after consulting data or calling a tool.

• Voice agents can be tested inside the ElevenLabs dashboard, embedded in a website, or connected to a phone number through Twilio. The tutorial focuses on the website option and uses Claude Code in VS Code, including plan mode, API-key setup, testing, iteration, and debugging a time-zone problem.