voice-eval
Evaluation harness for AI voice agents. Because 'sounds fine on the demo call' isn't a strategy.
§ 01 · What it is
An open-source evaluation harness that scores every call an AI voice agent handles — for latency, intent accuracy, escalation correctness, cost per call, and PII leakage. Point it at a recording, feed it the expected metadata, and get a pass/fail with actionable failure breakdowns. Built to catch drift before your customers notice.
§ 02 · Money it saves
§ 03 · How it works
- 01
Records every call with metadata: latency at each stage, model version, prompt version, confidence scores, cost, escalation decisions.
- 02
Compares each call against a rubric — pass thresholds for latency (< 1.5s), intent accuracy (> 95%), escalation correctness (100% for high-value bookings), and hallucination detection (LLM-as-judge scoring against grounded data).
- 03
Runs nightly against a held-out set of 20-50 golden calls so model updates or prompt changes get caught before they hit production.
- 04
Ships with a CLI + web dashboard. Exports Slack alerts when quality drops.
Stack
§ 04 · Why it beats what's out there
Every general-purpose LLM eval framework treats voice as an afterthought — you feed it text, it scores text. Voice has specific failure modes (per-stage latency, escalation correctness, PII in transcripts) that text-first tools miss.
| What's out there | The gap it leaves |
|---|---|
| General-purpose LLM eval platforms | Built text-first. Don't handle latency-at-each-pipeline-stage, escalation correctness, or PII leakage in voice transcripts — the three failure modes that actually matter for a voice agent in production. |
| Analytics baked into commercial voice-agent platforms | Vendor lock-in. If you switch your voice stack, you rebuild your evals from scratch. This runs against any provider — swap models or vendors freely and your evaluation history moves with you. |
| Manual review of random call samples | Doesn't scale past a few hundred calls a month. Catches quality drift after customers do, not before. Nightly automated eval runs catch drift the same night. |