What 'good' looks like in an AI voice agent — and why most fall short

If you've called any large company's helpline in the last two years, you've probably talked to an AI voice agent without realising it. The good ones sound close enough to human that you don't clock them in the first sentence. The bad ones make you press "0 for a human" within twenty seconds.

The difference between the two isn't the underlying AI — most people are using variants of the same models. The difference is the engineering around it. If you're considering replacing a receptionist line, missed-call handling, or after-hours cover with an AI voice agent, here's what actually separates the systems worth paying for from the ones you'll rip out in six months.

The number that decides everything: response latency

There's a single technical measurement that determines whether an AI voice agent feels human or annoying: the time between the caller finishing their sentence and the AI starting to respond.

• Under 1.5 seconds: feels like a person. Callers don't notice.
• 1.5 to 3 seconds: feels slightly off. Callers stop trusting it.
• Over 3 seconds: feels broken. Callers hang up or interrupt.

Most vendors will not tell you their real average. If they can't produce a number when asked, assume it's above two seconds — because they'd be quoting it otherwise. On the receptionist system I run for the demo on this site — Sophie at the fictional Blackbird Inn — the average is under 1.3 seconds. That's after months of engineering specifically to close that gap.

Why sub-second matters for your business

Beyond the "feels human" test, low latency changes what the AI can actually do for you:

• Bookings complete. A slow bot loses ~30% of callers to hang-up mid-conversation. A fast one converts them.
• After-hours coverage is real. If out-of-hours calls take five rings and 30 seconds to be answered by a bot, most callers won't wait. If the bot picks up in two rings and speaks like a person, they will.
• Complex requests don't derail. A slow bot has to keep saying "let me check on that" while it processes. A fast one can naturally handle multi-step requests like "actually can you make that a table of six instead, and add allergies for two of us."

The other things that separate good from bad

Latency isn't the only thing. Four other bars a voice agent should meet before you deploy it in front of customers:

1. It knows when to hand off to a human. Not every call is bookable by a bot. The right question isn't "can the AI handle this?" — it's "does the AI know when it can't?" A confidence threshold plus a clear escalation path (SMS to the manager's phone, ring the on-call number, take a message) is non-negotiable.

2. It doesn't lie. The most damaging failure mode for a voice agent isn't refusing to answer — it's making something up. Confirming a table for six people at a time you don't actually have. Quoting a price that's out of date. The system needs to be grounded in your real live data (bookings, menus, prices) and refuse to invent when it doesn't know.

3. Every call is logged and reviewable. If a customer complains that the bot promised them something, you need to be able to pull the recording and the exact conversation within seconds. Anything less and you're one bad call from a Google review disaster.

4. It handles UK accents and place names. Sounds obvious. Isn't. Most out-of-the-box voice tech is trained on American English and mangles Welsh place names, regional accents, and standard UK phrasing. Test any system with your actual customer base before signing.

What to ask a vendor before buying

Three questions:

• "What's the average response latency, measured end-to-end?" — if they don't have a number, walk away.
• "How does the bot decide to escalate to a human, and what happens when it does?" — vague answers here mean the bot will fumble things and you won't find out until a customer tells you.
• "Can I hear a recording of a real call it handled last week?" — the best proof isn't a slick demo, it's an unscripted call. If they can't share one (with client permission), they probably don't have any running in real production.

If you want to hear one yourself

Call Sophie on this site — it's a real voice agent, running the exact stack I ship for hospitality clients. You'll hear the latency, the natural back-and-forth, and the escalation logic first hand. Free, no signup, browser handles the mic.

If you're weighing up an AI voice agent for your business — reception cover, bookings, after-hours — I do a free 30-minute reality-check call. No sales pitch, just an honest read on whether it's worth it for your specific setup. Book here.