What callers actually hear on a voice AI agent
A caller gives their name. Five seconds of dead air pass before the bot answers. On a synthetic test dashboard, median latency looked fine at 1.3 seconds.
Bot logs only time what happens on your server. They miss network delay in both directions. They don't show whether the bot actually went quiet when a caller interrupted, or whether a crashed worker left someone sitting on a dead line.
I built a small restaurant reservation voice agent and tested it from the caller's side. A second program joins each call as a real caller, speaks five lines, records what comes back, and times every gap from the caller's ear.
Three problems showed up that server logs didn't catch.
1. Synthetic tests looked fine while real speech waited 4 to 5 seconds
Synthetic test calls can hide turn-detection bugs. On scripted calls, the agent answered in about 1.4 seconds, and lowering the silence timeout from 3 seconds to 1 second barely changed anything.
Real human speech broke that assumption. On a live call, the turn detector held several short answers until the full timeout, leaving 4 to 5 seconds of silence. Dropping the timeout to 1 second cut the worst wait in half, down to about 2.4 seconds. When a caller cut in mid-sentence, the bot stopped talking in about half a second.

2. Swapping to OpenAI's newer model slowed every reply
OpenAI retires gpt-4.1 on January 14, 2027 and lists gpt-6-luna as its replacement. Moving to the newer model isn't a free drop-in.
In head-to-head rounds on the same day, gpt-6-luna ran slower every single round, adding nearly half a second to every reply. One call in 9 also stalled, with one turn waiting over 10 seconds and another abandoned after 15 seconds. That's too few calls to give a stall rate, but gpt-4.1 had none on the same day. Testing from the caller's side catches regressions like this before customers hear them.

3. A mid-call crash left callers on a silent line for over two minutes
When a voice worker crashes mid-sentence, nothing inside that process is left to hang up the phone. Killing the bot mid-reply stranded the caller in dead silence for the full two minutes I watched, every time.
Running the bot under a small external watcher script that ends the room as soon as the process exits cut hang-up time from over two minutes down to about 0.3 seconds.

I run this check on other teams' voice agents as fixed-price work. If your callers hit gaps like these, write to hello@jdwilson.ca.
Technical reference: setup, full numbers, and limits
Full scripts and run summaries live in the repo: https://github.com/johnwils/voice-agent-caller-tests (opens in a new tab).
Setup
- Stack: Pipecat 1.15, Daily WebRTC rooms, Deepgram speech to text, Cartesia text to speech, Python 3.13.
- Caller harness: A headless daily-python client joins a fresh room per call, plays five recorded reservation lines in the macOS say voice, and records the incoming bot audio.
- Server-side comparison: Bot logs come from Pipecat's UserBotLatencyObserver.
- Lead config (Config C): Claude Haiku 5.5 (thinking off) with local smart-turn v3 (stop_secs 1.0 s).
Latency summary (8 scripted calls, 40 turns per config)
| Config | Model and turn rule | Caller median | Caller p95 | Caller max | Bot-side median |
|---|---|---|---|---|---|
| C (lead) | Claude Haiku 5.5, smart-turn 1.0 s | 1.35 s | 1.54 s | 1.56 s | 1.18 s |
| B | Claude Haiku 5.5, smart-turn 3.0 s | 1.33 s | 1.52 s | 1.56 s | 1.19 s |
| D | Claude Haiku 5.5, fixed 0.6 s timeout | 1.85 s | 2.03 s | 2.16 s | 1.64 s |
| A (Nov 10 baseline) | OpenAI gpt-4.1 (retires 2027-01-14), smart-turn 3.0 s | 1.17 s | 1.60 s | 1.78 s | 1.00 s |
| A (Nov 17 control) | OpenAI gpt-4.1, smart-turn 3.0 s | 1.21 s | 1.61 s | 2.08 s | 1.07 s |
| E (Nov 17) | OpenAI gpt-6-luna (effort low), smart-turn 3.0 s | 1.65 s | 2.25 s | 2.71 s | 1.52 s |
Notes on the A vs E model test and crash test
- Network gap: On Config C, caller-side median was 1.35 s vs 1.18 s in the bot log, so network delay in both directions added about 0.17 s.
- Model slowdown: Unrounded medians were 1.647 s (E) vs 1.214 s (A), a 0.433 s gap across all 8 rounds (by 0.31 to 0.64 s). Almost all of it sat in LLM time (0.92 s vs 0.50 s median), including a median 0.19 s of reasoning before the first word at effort low.
- Config E tail stall: One of 9 calls missed the 15 s reply limit, with LLM time to first output hitting 7.22 s on line 1 and 9.38 s on line 5. That failed call is excluded from the 8-call row above. Across all 45 turns, Config E's median is 1.68 s, p95 is 2.64 s, and max is 17.46 s.
- Barge-in (Config C, 5 calls): The caller spoke 1.0 s into a long reply. Bot audio stopped 0.50 to 0.57 s later from the caller's side and stayed silent for over 4 s.
- Bot crash (related to pipecat #6440): Sending SIGKILL 1.0 s into a reply left the caller in an empty room for 120 s (4 of 4 runs). Daily marked the bot gone as a normal leave after 33 to 36 s, which doesn't end the room. Running the bot under a small external watcher script calling Daily's REST eject endpoint ended the call in 0.25 to 0.48 s (median 0.28 s, 5 of 5 runs).
Limits
- The main latency, barge-in, and crash runs used a synthetic macOS say voice on one Mac and one network.
- Caller-side gaps include network delay in both directions and exclude browser audio delay.
- Barge-in was tested on Config C only (5 calls).
- Human-voice testing was one Chrome call per config for A to D (7 to 9 turns each, bot-side logs only) and none on Config E.
- 40 turns per config compares these setups on one machine and does not prove a p99.
- Config E was tested only at reasoning effort low on the default tier, and asked for a calendar date after “Tuesday” in 8 of 8 calls while Config A didn't.
- Tests used Daily WebRTC rooms only, with no phone leg (PSTN or SIP) yet.
- The watcher script only catches a process exit. A frozen bot that stays alive gets past it and needs a heartbeat.
more
- Sessionwire
- The product, year by year.
- Call rebuild
- How I rebuilt Sessionwire's WebRTC calls on Daily in seven releases, July 2026.
- Triage
- Read-only tools an agent uses to find why something broke in Sessionwire.
- Search
- CLAP, MERT and hybrid vector search for people and music, under $30 a month.
- Remote control
- Controlling someone else's computer in a call, over a WebRTC data channel.
- Firelight
- The pipeline behind a sleep-story channel.