How AI Voice Agents Work

Four parts, each usually from a different provider. Speech-to-text (STT) listens to the audio stream and produces a running transcript; it needs to be fast and to cope with accents, background noise and the caller talking over the agent. The language model (LLM) reads the transcript and the agent’s brief — who it is, what the call is for, what questions to ask, what it may not say — and produces the next reply; it can also call tools mid-conversation, such as checking a calendar. Text-to-speech (TTS) turns that reply into a voice; the choice of voice and provider decides how natural the agent sounds and which languages it can speak. Telephony connects all of this to the phone network: a SIP trunk from a carrier carries the call, and a media server handles the audio in real time.

A fifth piece sits around the others: the orchestration that streams audio in and out, decides when each component runs, records the transcript and outcome, and applies the campaign rules — calling windows, pace, retries, suppression lists.

Every reply has to wait for the transcript to finalise, the model to respond and the first audio to be synthesised. Each stage adds delay, and the caller experiences the sum as the pause before the agent speaks. In human conversation that pause is short; if the agent’s is noticeably longer, people talk over it, repeat themselves or hang up.

The levers are: streaming at every stage rather than waiting for complete outputs; a fast STT model; a language model chosen for response speed on conversational turns rather than raw capability; a TTS provider that starts audio quickly; and running the media server close to the telephony provider. Turn-detection settings trade latency against interruptions, which is why a well-built agent lets you tune them and test from the browser before any real call.

Three patterns account for most of the use. Cold and warm calling: dialling a verified list inside a window to confirm interest, ask qualification questions and offer a meeting — most effective when the list is good and the first sentence gives the person a reason to stay. Appointment setting and confirmation: booking a slot on the call, then calling again to confirm or reschedule, which is structured enough for an agent to do well. Inbound qualification: answering calls a small team would otherwise miss, collecting the essentials and either booking a callback or handing straight to a person.

What the agent should not do on a sales call is negotiate, promise or handle a serious complaint. The right design escalates those — “a colleague will follow up” — and flags the call, so the human salesperson picks up exactly where their judgement is needed.

Rules on automated calling, consent for recording, disclosure that the caller is speaking to an AI, calling hours and do-not-call registries vary by country and sometimes by state or sector. This guide cannot tell you what applies to you; check your jurisdiction, and the jurisdiction of the people you are calling, before running a campaign. A well-built platform gives you the controls to comply — suppression lists honoured on every campaign, calling windows, optional rather than default recording, and a configurable disclosure in the agent’s brief — but the responsibility for using them lawfully is yours.

Eligoo’s voice agent is Hook, an AI employee that runs on a self-hosted LiveKit stack: a media server, SIP for telephony, egress for optional audio recordings, and an agents worker that runs the conversation loop. You connect a SIP trunk from Twilio (created automatically from your Account SID and Auth Token), Telnyx, Plivo or Vobiz. Speech-to-text defaults to Deepgram; voices come from ElevenLabs, OpenAI or Deepgram; the language model comes from your connected OpenAI, Anthropic, Google Gemini, Groq or OpenRouter account. Language, behaviour presets, turn detection and interruption settings are configurable, and Indian languages are available through supported providers.

Outbound campaigns have a lead list, a calling window, a pace, daily roll-over and one line at a time; starting one is an approval. Inbound calls are dispatched to the same agent. Every call produces a transcript, an outcome and notes, with recording optional. A booked-meeting outcome creates a Google Calendar event with a Meet link when Google Workspace is connected, otherwise a task for Ledger. Pricing, promises, legal questions and serious complaints are outside Hook’s boundary and are flagged for a person.