What callers actually hear on a voice AI agent

Pipecat · Daily · Deepgram · Cartesia · Claude Haiku 5.5

A caller gives their name. Five seconds of dead air pass before the bot answers. On a synthetic test dashboard, median latency looked fine at 1.3 seconds.

Bot logs only time what happens on your server. They miss network delay in both directions. They don't show whether the bot actually went quiet when a caller interrupted, or whether a crashed worker left someone sitting on a dead line.

I built a small restaurant reservation voice agent and tested it from the caller's side. A second program joins each call as a real caller, speaks five lines, records what comes back, and times every gap from the caller's ear.

Three problems showed up that server logs didn't catch.

1. Synthetic tests looked fine while real speech waited 4 to 5 seconds

Synthetic test calls can hide turn-detection bugs. On scripted calls, the agent answered in about 1.4 seconds, and lowering the silence timeout from 3 seconds to 1 second barely changed anything.

Real human speech broke that assumption. On a live call, the turn detector held several short answers until the full timeout, leaving 4 to 5 seconds of silence. Dropping the timeout to 1 second cut the worst wait in half, down to about 2.4 seconds. When a caller cut in mid-sentence, the bot stopped talking in about half a second.

Synthetic script vs. real human call on Claude Haiku 5.5. The synthetic median is 1.33 s with a 3.0 s timeout and 1.35 s with a 1.0 s timeout. The worst wait on a real human call is 5.04 s with a 3.0 s timeout and 2.37 s with a 1.0 s timeout.

2. Swapping to OpenAI's newer model slowed every reply

OpenAI retires gpt-4.1 on January 14, 2027 and lists gpt-6-luna as its replacement. Moving to the newer model isn't a free drop-in.

In head-to-head rounds on the same day, gpt-6-luna ran slower every single round, adding nearly half a second to every reply. One call in 9 also stalled, with one turn waiting over 10 seconds and another abandoned after 15 seconds. That's too few calls to give a stall rate, but gpt-4.1 had none on the same day. Testing from the caller's side catches regressions like this before customers hear them.

Server logs vs. caller-side reply time across gpt-4.1, Haiku 5.5, and gpt-6-luna. The caller-side median is 1.21 s for gpt-4.1, 1.35 s for Claude Haiku 5.5 and 1.65 s for gpt-6-luna. gpt-6-luna was 0.43 s slower in 8 of 8 rounds, and 1 of 9 calls stalled.

3. A mid-call crash left callers on a silent line for over two minutes

When a voice worker crashes mid-sentence, nothing inside that process is left to hang up the phone. Killing the bot mid-reply stranded the caller in dead silence for the full two minutes I watched, every time.

Running the bot under a small external watcher script that ends the room as soon as the process exits cut hang-up time from over two minutes down to about 0.3 seconds.

Time a caller sits on a silent line after a mid-call bot crash. With the bot on its own, the line was still silent at 120 s in 4 of 4 runs. With an external watcher, the call ended in 0.28 s median in 5 of 5 runs.

I run this check on other teams' voice agents as fixed-price work. If your callers hit gaps like these, write to hello@jdwilson.ca.

Technical reference: setup, full numbers, and limits

Full scripts and run summaries live in the repo: https://github.com/johnwils/voice-agent-caller-tests (opens in a new tab).

Setup

  • Stack: Pipecat 1.15, Daily WebRTC rooms, Deepgram speech to text, Cartesia text to speech, Python 3.13.
  • Caller harness: A headless daily-python client joins a fresh room per call, plays five recorded reservation lines in the macOS say voice, and records the incoming bot audio.
  • Server-side comparison: Bot logs come from Pipecat's UserBotLatencyObserver.
  • Lead config (Config C): Claude Haiku 5.5 (thinking off) with local smart-turn v3 (stop_secs 1.0 s).

Latency summary (8 scripted calls, 40 turns per config)

ConfigModel and turn ruleCaller medianCaller p95Caller maxBot-side median
C (lead)Claude Haiku 5.5, smart-turn 1.0 s1.35 s1.54 s1.56 s1.18 s
BClaude Haiku 5.5, smart-turn 3.0 s1.33 s1.52 s1.56 s1.19 s
DClaude Haiku 5.5, fixed 0.6 s timeout1.85 s2.03 s2.16 s1.64 s
A (Nov 10 baseline)OpenAI gpt-4.1 (retires 2027-01-14), smart-turn 3.0 s1.17 s1.60 s1.78 s1.00 s
A (Nov 17 control)OpenAI gpt-4.1, smart-turn 3.0 s1.21 s1.61 s2.08 s1.07 s
E (Nov 17)OpenAI gpt-6-luna (effort low), smart-turn 3.0 s1.65 s2.25 s2.71 s1.52 s

Notes on the A vs E model test and crash test

  • Network gap: On Config C, caller-side median was 1.35 s vs 1.18 s in the bot log, so network delay in both directions added about 0.17 s.
  • Model slowdown: Unrounded medians were 1.647 s (E) vs 1.214 s (A), a 0.433 s gap across all 8 rounds (by 0.31 to 0.64 s). Almost all of it sat in LLM time (0.92 s vs 0.50 s median), including a median 0.19 s of reasoning before the first word at effort low.
  • Config E tail stall: One of 9 calls missed the 15 s reply limit, with LLM time to first output hitting 7.22 s on line 1 and 9.38 s on line 5. That failed call is excluded from the 8-call row above. Across all 45 turns, Config E's median is 1.68 s, p95 is 2.64 s, and max is 17.46 s.
  • Barge-in (Config C, 5 calls): The caller spoke 1.0 s into a long reply. Bot audio stopped 0.50 to 0.57 s later from the caller's side and stayed silent for over 4 s.
  • Bot crash (related to pipecat #6440): Sending SIGKILL 1.0 s into a reply left the caller in an empty room for 120 s (4 of 4 runs). Daily marked the bot gone as a normal leave after 33 to 36 s, which doesn't end the room. Running the bot under a small external watcher script calling Daily's REST eject endpoint ended the call in 0.25 to 0.48 s (median 0.28 s, 5 of 5 runs).

Limits

  • The main latency, barge-in, and crash runs used a synthetic macOS say voice on one Mac and one network.
  • Caller-side gaps include network delay in both directions and exclude browser audio delay.
  • Barge-in was tested on Config C only (5 calls).
  • Human-voice testing was one Chrome call per config for A to D (7 to 9 turns each, bot-side logs only) and none on Config E.
  • 40 turns per config compares these setups on one machine and does not prove a p99.
  • Config E was tested only at reasoning effort low on the default tier, and asked for a calendar date after “Tuesday” in 8 of 8 calls while Config A didn't.
  • Tests used Daily WebRTC rooms only, with no phone leg (PSTN or SIP) yet.
  • The watcher script only catches a process exit. A frozen bot that stays alive gets past it and needs a heartbeat.

more

Sessionwire
The product, year by year.
Call rebuild
How I rebuilt Sessionwire's WebRTC calls on Daily in seven releases, July 2026.
Triage
Read-only tools an agent uses to find why something broke in Sessionwire.
Search
CLAP, MERT and hybrid vector search for people and music, under $30 a month.
Remote control
Controlling someone else's computer in a call, over a WebRTC data channel.
Firelight
The pipeline behind a sleep-story channel.