DuckyCutz Voice-Eval Harness — how it works, how to drive it

DuckyCutz · Lesson 01 · Voice-eval harness

The robot mystery shopper that grades your phone agent

How the DuckyCutz Thai voice-eval harness works end to end, and how you drive it yourself, without spending money or trusting your ear alone.

Reading time ~9 min · Grounded in ADR DC-0011 (the recipe + mindset) and ADR DC-0010 (campaign #1's question) · Ask me anything unclear — I'm your teacher, not just this page.

01The problem: you were the whole test

Until now, checking whether the AI receptionist was good meant you picked up the phone, talked to it, and decided by ear if it felt right. That is a test of one person, on one call, with no written scorecard. Change the agent tomorrow and you can't say whether it got better, because there's nothing to compare against. It feels like testing. It isn't data.

The harness replaces you-with-a-phone with a machine that runs the same hard calls the same way every time, and writes down a score you can trust.

02What a "voice eval" actually is

The core analogy

A robot mystery shopper. One AI plays a difficult customer from a script and phones your agent. Another AI plays the examiner: it reads the whole conversation afterward and ticks pass/fail boxes on a checklist you wrote in advance.

The "phone call" isn't a real phone call. The robot caller connects to the agent over the internet (a websocket) instead of a phone line, so there's no per-minute telephone charge. But it's still real audio: the agent has to hear speech, turn it into text, think, and talk back, with real timing.

That last part is the whole point. We already have a cheaper test called /chat that types messages to the agent as text. Cheap text tests catch brain problems (wrong logic, a broken prompt). They can never catch ear-and-mouth problems, because no audio ever happens:

The ear (speech-to-text): did it hear a mumbled Thai name right?
The mouth & timing: did it talk over the caller, or leave an awkward gap?
The Thai filler: when it pauses to look something up, does it say "สักครู่นะคะ" and not a jarring English "One moment"?

Voice evals are the only test that exercises the ear and the mouth.

03Two stages: cheap gate, then paid ruler

Every campaign runs in two stages, cheapest first. Nothing reaches the expensive stage until it has earned it.

1 · Chat gate

Run the whole scenario library as text. Catches broken logic and prompt regressions before any money is spent.

≈ near-free

→

2 · Voice ruler

Only what passed chat earns real billed calls: AI brain + speech-to-text + voice + platform.

≈ $0.15–0.30 / call

Think of it as a written exam before the driving test. Fail the cheap written part and you never waste a driving examiner's afternoon. Passing chat says nothing about audio, so it's a gate, not a verdict.

04The 5 gates the examiner checks

The examiner is a machine ticking five boxes. All five are automated. You only get pulled in when they conflict or something looks off, never to grade every call by hand.

1Validity — was this even a fair test?If the robot customer broke character (started saying "I'm an AI, call the shop"), the call is thrown out. It counts for neither side.
2Task correctness — did it get the job right?Right name, right date and time, right booking action with the right details, questions actually answered.
3Pass rate — is it reliable, not lucky?Each hard scenario runs 3+ times. One good call is a story; a pass rate across several is data.
4Thai filler — does it stall in Thai?On every "let me check" pause it must play the Thai filler, never the English one. This is the exact bug campaign #1 is chasing.
5Latency — comparable pauses?The gap between "caller stops talking" and "agent starts talking." Compared only within the same campaign — never against a fixed number, because the raw figures VAPI reports are unreliable.

05The one hard rule: sandbox only, forever

Learned the hard way · 2026-07-13

We ran a test on a copy of the live agent. The copy still carried the real booking tools, so the robot's practice calls hit the real backend: they created a stray customer contact, polluted the live usage dashboard, and made a real booking you had to phone up and cancel.

The fix is deliberately boring: tests run only on a separate sandbox account whose tools point at a fake stand-in (vapi-test-relay.vercel.app) that does nothing real. If the sandbox runs low on credit, the answer is top it up with money, never to reach back into the live system "carefully." A test that can touch anything real is not a test, it's a live call.

06The mindset (the part that makes results trustworthy)

Test on the mess, not the clean. Every agent passes "book tomorrow 2pm, name Ruj." Models only differ on garbled real calls, so the scenarios are real messy transcripts.
Change one thing per run. The model OR the speech-to-text OR the prompt. Change two and you learn nothing about which one moved the score.
Write "pass" down before the call. The checklist is fixed first. Your ear is a tie-breaker, not the ruler.
The test set is the asset; models are throwaway. The durable thing you own is the growing library of hard calls, not "this model vs that one."
Every prod failure becomes a permanent test. When the live agent messes up, that exact call joins the library. The suite only ever gets meaner.

07The go-forward: run lean, not the whole gauntlet

The old instinct was to run all 8 scenarios across 2 options every time — about $3 a campaign and slow. New direction, and this is how you should drive it from now on:

The lean rule

Keep the full library on the shelf. For any given change, run only the 2–3 harshest scenarios that stress the exact thing you changed. Testing a speech-to-text swap? Run the mumbled-name and date-correction scenarios; skip the clean easy ones. A / B / decide, then stop. That drops a campaign to well under $1.

Deciding which scenarios and whether the result is good enough is what a /grilling session is for: it interrogates the plan before you spend, and keeps the docs current after.

08Where campaign #1 actually stands (the honest bit)

The first real campaign is testing two ways of hearing Thai: Azure vs ElevenLabs "Scribe" speech-to-text, with the agent's brain held fixed on gpt-5.4-nano. It is mid-flight and has no trustworthy verdict yet, for two reasons that are both faults in the test rig, not the agent:

Rig bug A · dedup

I asked for 8 scenarios × 3 attempts = 24 calls. VAPI saw the repeated IDs as duplicates and collapsed them to 8 — so each scenario ran once, not three times. That breaks gate #3 (the 3+ rule). Fix: give each attempt its own distinct object or separate run.

Rig bug B · grading stall

4 of the 8 calls came back "Simulation execution stalled after the target call had already ended." That's VAPI's examiner hanging after the call finished — a grading glitch on their side, not the agent failing. Don't read it as a bad agent.

The Azure half is done; the Scribe half is still running. Bottom line: no verdict yet, and the rig needs those two fixes before campaign #1's numbers mean anything.

Cheat sheet: how you drive it

Name the one change you want to prove (e.g. "does Scribe hear Thai names better than Azure?"). One variable only.
Pick 2–3 harsh scenarios that stress that change. Ignore the clean ones.
Run the chat gate first (near-free). If it fails there, stop — no audio needed.
Run the voice ruler on the survivors, 3+ attempts each, in the sandbox.
Read the 5 gates. If two conflict, or you see a stall / dedup, treat it as a rig problem, not a verdict.
Decide and stop. Cheapest option that passes wins. Fold the result back into the ADR.

Check yourself (answers hidden — say it out loud first)

Why can't the cheap chat test replace the voice test?

Chat is text only, so no audio ever happens. It catches brain/logic bugs but never the ear (speech-to-text), the mouth/timing, or the Thai-vs-English filler. Those are voice-only truths.

The examiner says 4 calls "stalled after the call ended." Bad agent?

No. That's the grader hanging on VAPI's side after the call finished — a rig bug, not an agent failure. Don't count it against the agent.

Sandbox is low on credit. Can you point one quick test at the live system?

Never. Top up the sandbox with money. Anything touching the real backend can create real contacts and real bookings — that's the exact mistake from 2026-07-13.

You're testing a new speech-to-text. Run the whole 8-scenario library?

No — run lean. Pick the 2–3 harshest scenarios that stress hearing (mumbled name, date correction), skip the clean smoke ones, A/B/decide, stop. Under $1.