Short answer: almost nobody. We went looking for prior art on an idea we thought was ours, and found that the machinery to test AI conversations properly already exists — it's just pointed at the wrong thing.
Here's the short version. If you build an AI agent that talks to customers, there is now a whole industry of tools that will hammer it with thousands of synthetic calls, score every one, tell you what to change, and block your release until it passes. If you build an AI simulator that talks to your new hires, there is nothing. Your trainees are the test.
We didn't set out to write that. We set out to check whether we'd invented anything.
We've been building a testing layer for our own training simulators — a lab where a synthetic AI caller dials our own simulations, thousands of times, and an independent AI evaluator scores whether the simulated customer actually behaved like a customer. Failures trigger changes to the prompts, personas and scenarios. The suite reruns. Nothing reaches a human until it passes.
Before going further, we did the thing you're supposed to do: we tried to prove it wasn't new. Academic papers, patent databases, and every commercial product in the category we could find. Some of what we found was uncomfortable. That's the point of the exercise.
The four-stage pattern — synthetic testing, automated evaluation, automatic rebuild, release gate — is well established. For AI agents.
Hamming AI, which builds QA tooling for voice agents, runs 50,000+ concurrent synthetic test calls with accents, background noise and interruptions, scores them with automated judges, turns production failures into test cases and recommends prompt changes, and runs regression suites in CI/CD that block deploys when thresholds are breached. They're explicit about why: a test that passed yesterday can fail today because model weights shifted, not because anyone changed the code.
Cresta describes something similar for their AI agents — simulated visitors spinning up hundreds of scenario variations for regression tests and drift checks, and a release process that validates every update against known edge cases with rollback safeguards.
That's rigorous engineering. It's also completely normal now. Nobody ships a customer-facing voice agent on vibes any more.
Then look at what happens on the training side of the same industry.
Cresta launched a Training Simulator for human agents in July 2026, built on the same AI agent technology. In their own announcement, the quality step is that scenarios can be validated against live quality criteria before publishing. That's a person checking a scenario. Sensible — but it's not the same machinery they point at their customer-facing agents.
Solidroad markets a closed loop, and it is one, but a different one: QA findings from live conversations automatically trigger personalised training simulations. The loop optimises the human. Nothing in it tests the simulator.
Zenarate's methodology is genuinely good pedagogy — real-time coaching when a learner misses a required practice, scored on intent rather than keywords. ReflexAI scores 100% of conversations against custom frameworks and validates responses against your own policy documents. Second Nature's own guidance on building good scenarios is to put together a team with an operations SME and a training leader.
Every one of those is a reasonable answer. Every one of them puts a human in the position of finding out whether the simulation works.
Being straight about it: we didn't find a single published system — commercial, academic or patented — that runs synthetic testing, automated evaluation, automatic rebuild and a release gate as one connected pipeline whose output is a training product for humans.
The nearest academic work is TeachTune, presented at CHI 2025, where teachers create simulated students and review AI tutors by watching automated conversations between them. It stops where you'd expect: the teacher reads the transcripts and revises by hand. There's a formal “quality gate” paper from 2026 that does the full loop with rollback grading — but the thing under test is a marketing AI, not a training simulation.
The patent landscape has plenty of adjacent claims. Simulated customers testing service bots. Customer simulations coaching human agents. Nobody, as far as we can see, has connected the two.
And the idea underneath is older than any of us. The founder of the AI-in-education society was arguing in the 1990s that simulated students should work as wind tunnels for testing tutoring system designs. The concept has been sitting there for thirty years waiting for the models to catch up.
Forget the architecture for a second. Here's the practical version.
You buy an AI training simulator. It works well for six months. Then the vendor upgrades the language model underneath it, or swaps the voice provider, because everyone does, constantly. The simulated angry customer becomes 15% more agreeable. Nobody notices.
Your agents now train against a customer who is easier than the real thing. They pass. They hit the floor. They struggle. The training scores look fine, so nobody connects it back. You find out through attrition and CSAT, months later, if you find out at all.
That's not a hypothetical failure mode. It's the exact reason the voice-agent QA industry exists — Hamming's own material makes the point that prompt and model changes cause silent behavioural drift rather than obvious breakage.
Five questions worth asking any vendor — including us
When you change the model underneath, what reruns before my agents see it?
Who or what scores whether your simulated customer is still realistic?
Can you show me a version history of a scenario and what changed and why?
What's your pass threshold, and what happens to a build that fails it?
When last did a build of yours fail and get pulled?
Question five is the honest one. Any real gate has failed something. If nothing has ever failed, there's no gate.
We've been building this as the Synthetic Voice Lab — synthetic trainees with expected-score envelopes, an audio-native evaluator, and a per-release scorecard that has to pass before anything ships to a human. It has caught real problems. It has also blocked our own releases more than once, which is the part that tells us it's working.
We're not claiming we invented the loop. The prior art is clear that we didn't. What we found is that in this industry, the loop gets pointed at the AI and not at the training — and we think that's backwards, because the cost of a bad AI agent is one bad call, and the cost of a bad training simulator is a cohort of agents who were confidently prepared for the wrong job.
See the simulation itself
Our product demos are live — experience the synthetic voice lab in action.
Sources
All findings above come from publicly published material, checked in August 2026. No vendor named here was contacted, and nothing above is a criticism of their products. Every one of them is doing real work. We're describing what is and isn't publicly documented, nothing more.