An AI voice agent is a system that listens to a user's speech, reasons about it with a large language model, and replies in a natural-sounding voice in…
See the two architectures that turn speech in into speech out.
An AI voice agent adds ears and a mouth to an LLM: it hears speech, understands and reasons, and speaks back. There are two ways to build it. The classic pipeline chains three services — speech-to-text (STT) transcribes the audio, the LLM decides the reply, and text-to-speech (TTS) voices it. A newer approach uses a single real-time speech-to-speech model that takes audio in and emits audio out.
The pipeline gives you control and easy swapping of each part; the speech-to-speech model is simpler and lower-latency but less modular. Many production agents still use the pipeline for its flexibility.
Follow the numbered steps to assemble a working voice loop.
A pipeline voice agent takes four steps. (1) Capture microphone audio and stream it to an STT service. (2) Detect the end of the user's turn and send the transcript to the LLM, ideally with the conversation history. (3) Stream the LLM's text to a TTS service as it generates. (4) Play the audio, and stop it immediately if the user starts speaking again.
Streaming at every stage is what makes it feel live: start transcribing while the user talks, start speaking while the model generates.
history = [] while call_active: audio = mic.listen_until_silence() # 1: capture + turn detection text = stt.transcribe(audio) # 2: speech -> text history.append({"role": "user", "content": text}) for chunk in llm.stream(history): # 3: reason, stream tokens tts.speak(chunk) # 4: text -> speech, as it arrives history.append(assistant_reply)
One turn: capture audio and detect the pause, transcribe, stream the LLM reply into TTS so speech starts before the full answer exists, and keep history so the agent has context. Real implementations run these concurrently and interrupt speak() on barge-in.
Handle timing and interruptions — where most voice agents break.
The components are the easy part; natural conversation is the hard part. Turn detection decides when the user has actually finished, not just paused mid-thought — end too early and you interrupt them, too late and the agent feels sluggish. Semantic turn detection uses the words, not just silence, to judge.
Barge-in lets the user cut in while the agent is speaking: the moment their voice is detected, the agent stops talking and starts listening. Aim for roughly an 800-millisecond response latency end to end; beyond about a second, the pause feels unnatural, which is why streaming and fast models matter.
Avoid the errors that make a voice assistant feel robotic or rude.
The usual failures are about timing, not intelligence. No streaming means the user waits for the whole answer before hearing anything — stream every stage. Silence-only turn detection cuts people off mid-sentence — use semantic turn detection. No barge-in makes the agent talk over the user — always allow interruption.
Also keep replies short (spoken text should be conversational, not an essay), handle transcription errors gracefully, and design for noisy audio and accents. Test with real speech, not typed input.
An AI voice agent hears speech, reasons with an LLM, and speaks back in real time, built as an STT to LLM to TTS pipeline or a single speech-to-speech model. Stream every stage so the reply starts fast. The real work is the conversation layer: semantic turn detection to know when the user is done, barge-in to allow interruption, and an ~800ms latency budget. Most failures are timing bugs, not intelligence gaps.
Sketch a voice assistant for booking appointments. Choose the pipeline or speech-to-speech architecture, describe how you would detect the end of a user's turn, and name the one interruption behavior (barge-in) you would implement first to make it feel human.
What is an AI voice agent?
A voice agent adds speech input and output around an LLM so a user can talk to it and hear a spoken reply.
What are the main components of a voice assistant?
The classic pipeline chains STT, an LLM, and TTS; an alternative is one speech-to-speech model that handles audio end to end.
What is barge-in in a voice agent?
Barge-in makes conversation natural by halting the agent's speech the moment the user starts talking.
What latency should a voice agent aim for?
Sub-second response is the target for natural turn-taking, achieved by streaming STT, LLM, and TTS rather than waiting for each to finish.