How to Build an AI Voice Agent for Inbound Sales Calls: A Twilio + GPT-4 Step-by-Step Guide

Why a basic IVR menu loses more leads than it saves

A traditional IVR forces the caller through 'press 1 for sales, press 2 for support' menus that assume the caller already knows your org chart. Most callers hang up during the menu, especially on mobile, where each layer of the tree adds friction the caller did not sign up for.

Voicemail is not much better: a large share of inbound sales callers who hit voicemail do not leave a message, and even when they do, the average callback happens hours later, by which point the caller has often already spoken to a competitor. The fix is not a smarter menu, it is a system that can actually converse: understand the caller's intent from the first sentence, answer common questions, and route or book a meeting without the caller waiting on hold.

The architecture: what you are building

• Twilio phone number configured with a Voice webhook that returns TwiML starting a bidirectional Media Stream over WebSocket, instead of static <Say> and <Gather> prompts.
• A small WebSocket server (Node or Python) that receives the raw audio stream (mulaw, 8kHz) from Twilio and forwards it to a streaming speech-to-text API.
• Streaming STT, Deepgram's streaming API or OpenAI's real-time transcription, which returns partial and final transcripts with roughly 200-300ms latency instead of waiting for the caller to finish speaking.
• GPT-4 (or GPT-4o for lower latency) as the conversation engine, called with function definitions for check_calendar_availability, book_meeting, and transfer_to_human, so it can take real actions, not just talk.
• Low-latency text-to-speech, ElevenLabs Turbo or a neural voice through Twilio's own <Say>, streamed back into the same Media Stream so the caller hears a response without a long silent gap.
• A session store keyed by call SID that holds the running transcript and any lead data collected mid-call, plus a call log written to your CRM once the call ends.

Step-by-step: building the voice agent

• Step 1 — Provision the number and webhook: buy a Twilio number, set its Voice webhook to your server's endpoint, and return TwiML with a <Connect><Stream> verb pointing at your WebSocket URL instead of a static prompt tree.
• Step 2 — Handle the media stream: on your WebSocket server, accept the connection, buffer incoming mulaw audio chunks, and pipe them to your streaming STT provider's socket as they arrive, rather than batching and sending after the caller stops talking.
• Step 3 — Detect end of turn: use the STT provider's endpointing signal (Deepgram exposes an is_final flag with a configurable silence threshold, typically 400-700ms) to decide when the caller has finished a thought, so the agent responds promptly without cutting them off mid-sentence.
• Step 4 — Run the GPT-4 turn: send the finalized transcript segment plus recent conversation history to GPT-4 with a system prompt defining the agent's role, tone and the exact information it must collect (name, company, use case, timeline), and register your functions so the model can call them instead of only generating text.
• Step 5 — Handle function calls: when GPT-4 returns a function call for check_calendar_availability, query your calendar API (Google Calendar or Cal.com) server-side, return the result to the model, and let it continue the conversation with real availability instead of guessing times.
• Step 6 — Synthesize and stream the reply: send the model's text response to your TTS API in streaming mode, and forward the returned audio chunks back into the Twilio Media Stream as they arrive, so playback starts within a few hundred milliseconds instead of waiting for the full sentence to render.
• Step 7 — Support barge-in: keep listening to the caller's audio stream while your agent is speaking, and if new speech is detected, stop the outbound audio immediately and process the new input, since callers frequently interrupt and a voice agent that talks over them feels broken immediately.
• Step 8 — Book the meeting and log the call: once the model calls book_meeting with a confirmed slot, write the event to your calendar, push the full transcript and structured lead data to your CRM, and end the call with a clear confirmation of what was booked.

Common failure points and how to fix them

• Latency stacking across the pipeline: STT, GPT-4 and TTS each add their own delay, and if they run in sequence per turn, total round-trip latency can exceed 2-3 seconds, which feels broken on a phone call. Fix by streaming at every stage, start TTS on the first sentence of the model's response instead of waiting for the full completion.
• The agent mishearing names, company names or numbers: general STT models struggle with proper nouns and phone numbers spoken quickly. Fix by passing a custom vocabulary or keyword-boosting list to your STT provider with your product name, common company names in your market, and by having the agent read back critical details ('so that's a callback at 98765, correct?') before confirming.
• Dead air when a function call takes too long: a calendar API lookup that takes 2-3 seconds leaves silence on the line, which callers read as a dropped call. Fix with a short filler response queued immediately ('let me check that for you') while the function call runs in parallel.
• The agent not knowing when to give up and transfer: an ungrounded model will keep trying to answer a question it cannot handle. Fix by giving the model an explicit transfer_to_human function and instructing it in the system prompt to call it after two failed attempts to resolve the caller's request, routing to a live agent via Twilio's <Dial> verb with the transcript passed along.

How AIBOOTSTRAPPER helps

A voice agent that handles a clean demo call and one that survives real inbound volume, noisy lines, interruptions, accents and edge-case questions, are different engineering problems. The second needs tuned endpointing, a fallback strategy for every function call, and monitoring so a silent failure does not mean a day of missed calls with no alert.

If you would rather have this built and maintained for you, AIBOOTSTRAPPER's AI automation team designs custom voice agents around your actual sales script and call volume, integrated with your real calendar and CRM, and can pair it with a cloned voice so the agent sounds consistent with your brand.