Skip to content

Voice and realtime agents

  • Agents
  • Specialist topic or differentiator
  • About 1 h 30 min to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

A voice agent is an agent loop with a clock attached: the caller hears every millisecond of STT, LLM and TTS stacked end to end, and starts reading pauses over about 800 ms as hesitation. Interviews ask you to choose between a chained STT→LLM→TTS pipeline (inspectable, swappable stages, transcripts for free) and a speech-to-speech model (lower latency, less control), then to defend a per-stage latency budget and explain what time-to-first-audio measures. The second cluster of questions is turn-taking: why a fixed silence timeout cuts people off, what a VAD can and cannot hear, when a semantic end-of-turn model earns its compute, and how barge-in differs from end-of-turn. Expect follow-ups on transport (WebRTC for browsers, WebSocket server-side, SIP for phones), calling tools without going silent, handing off to a specialist agent or a human, and how you would evaluate the thing with synthetic callers before real ones.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Compare chained and speech-to-speech designs and name two reasons to pick each
  • Define the latency number a caller actually feels and say why p50 and p95 are reported separately
  • Explain how tool calls and handoffs attach to a realtime session without the conversation stalling
  • Sketch a crawl-walk-run evaluation plan that ends with simulated multi-turn callers
  • Explain why a fixed silence timeout interrupts people who are still thinking
  • State what a VAD detects and give one pause it will misread as a turn end
  • Describe how a semantic end-of-turn model uses the transcript and what it costs
  • Distinguish barge-in from end-of-turn and name the different cost of a mistake in each
  • Define false-interruption rate and say how you'd measure it on real calls
  • Quote the rough human turn-transition gap and the point where callers notice lag

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Voice agents (opens in a new tab)

    OpenAI API docs · OpenAI · 30 min · Free · checked 21 Oct 2026

    Why
    It frames the exact design question interviewers open with (chained pipeline vs speech-to-speech vs full-duplex) and then tells you how to measure latency and evaluate the agent.
    What to read
    Read “Choose the right architecture”, skim “Build a speech-to-speech voice agent” and “Build a chained voice workflow” for the shape of each, then read “Evaluate your voice agent” including “Measure latency” carefully.
    What to skip
    The SDK code blocks and “Build a full-duplex voice agent” unless you're targeting a voice-platform role; “Next steps”.

  2. Free alternative

    Turn detection in voice agents: silence timeouts vs VAD vs semantic models (opens in a new tab)

    Deepgram Learn · Jose Nicholas Francisco · 25 min · Free · checked 21 Oct 2026

    Why
    Endpointing is the part of voice agents that candidates hand-wave; this gives you the three mechanisms, their failure modes, and a metric to argue with.
    What to read
    Read “Mechanism comparison at a glance” through “Fusing both signals in one model”, then “Measuring your false-interruption rate” and “Eager end-of-turn: 150-250ms earlier, and the cost”. In the FAQ, read “Is barge-in detection the same problem as end of turn detection?”.
    What to skip
    “Set your thresholds before you ship” and the rest of the FAQ unless you're tuning a real deployment.

  3. Go deeper

    Voice AI latency: why pauses read as hesitation (opens in a new tab)

    Deepgram Learn · Jose Nicholas Francisco · 25 min · Free · checked 21 Oct 2026

    Why
    Gives you the numbers interviewers want to hear: human turn gaps, where lag becomes noticeable, and a per-use-case budget to split across STT, LLM and TTS.
    What to read
    Read “When your agent should start speaking”, “The only delay your caller notices” and “Voice AI latency budgets by use case”, then “Set a budget, then measure against it”.
    What to skip
    “What a pause tells the person on the other end” (background) and the FAQ beyond “What if the LLM is the bottleneck instead of the speech layer?”.

Companies that test Agents most

Put Voice and realtime agents in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan