Building voice agents

Why AI voice agents fail, and how to test, measure and supervise them so they don't

The short answer

Why AI voice agents fail is rarely the technology alone. They are given calls they can't resolve, built as rigid flows, measured on containment, launched without an owner and never tested on difficult callers. The fix: one clear job, practice calls with hard callers before launch, pass marks, supervision of every call, and a full retest after every change.

By , founder and CEO of ShiftMateUpdated 16 min read
Scoring a practice call to find out why AI voice agents fail
Mike Steenkamp, founder and CEO of ShiftMate, a South African voice agent specialist
“ShiftMate is a South African voice agent specialist, and we've watched our own agents fail in testing so real callers don't have to. This guide covers why AI voice agents fail with real customers, from slow replies to AI customers that give in too easily to promises the system never kept, and how to test for each.”
Mike Steenkamp, founder and CEO of ShiftMate, Umhlanga, South Africa
Key takeaways
  • Most of why AI voice agents fail is decided before launch: the wrong job, the wrong target and no testing on difficult callers. Our complete guide to AI voice agents covers the basics.
  • In a Gartner survey of 187 service leaders (2024), 44% were exploring a voicebot and 11% piloting one, but only 5% had one deployed. Getting past the pilot is the hard part.
  • Test before launch with a library of at least 20 difficult practice calls, and make every version pass each one three times, not once.
  • Measure resolution, repeat calls and handover quality, not containment. Containment rewards an agent for keeping calls it should pass on.
  • When AI takes the easy calls, your people get the hard ones. One large operator in South Africa stretched agent training from an average of three weeks (2024/25) to eight (2025/26) because of AI.

Why AI voice agents fail: the short answer

AI voice agents fail on real people, not in demos. In a demo the caller is friendly, speaks clearly in a quiet room and asks the question the agent was built for. Real callers interrupt, raise two problems at once, get upset, have accents the speech recognition rarely hears, and ask for things nobody wrote down.

So the honest answer to why AI voice agents fail is that most failures are planted before launch: a job the agent can't finish, a flowchart instead of a conversation, a number that rewards the wrong behaviour, and no proof that it can handle the calls it will really get. The technology has improved quickly; the way businesses aim, test and supervise their agents has not kept up.

In a Gartner survey of 187 customer service leaders in mid-2024, 44% were exploring a customer-facing voicebot and 11% were piloting one, but only 5% had one deployed. This page explains why AI voice agents fail between pilot and launch, and gives you a test plan, a scorecard and a supervision routine from our own testing.

What are the five root causes McKinsey found, and what do they leave out?

McKinsey's operations team gave the clearest account so far of why AI voice agents fail, in a June 2026 article (When AI voice agents struggle, and how to raise them better). Their diagnosis: agents are given jobs they can't yet do reliably, because ambition runs ahead of capability and operations. They describe a fintech that automated all its customer service and saw satisfaction fall, and a healthcare organisation weighing a supplier's plan to automate 80 to 90% of interactions within 24 months, designed only for simple, happy-path calls. Their five causes, in our own words:

1

Asking how much to deflect, not what to resolve

"How many calls can we keep from people?" points the agent at everything. Ask instead which calls it can finish properly, and which it shouldn't touch yet.

2

A flowchart where a conversation should be

Real speech squashed into fixed paths breaks at the first interruption, change of mind or second problem.

3

A voice agent bolted on, with no owner

Weak links to the systems it needs, no clean handover to a person, and nobody responsible for tuning it after launch.

4

The wrong measure of success

Containment and cost per contact reward the agent for holding calls. Repeat calls rise and satisfaction drops.

5

Callers who already distrust voice bots

After years of bad phone menus, an agent that's slow or vague about what it can and can't do loses people quickly.

We agree with McKinsey's remedies, which read like a list of AI voice agent best practices: aim for resolution, design for real conversation, connect the agent to your systems first, and reward real improvement rather than containment. It also recommends shadowing live calls and running pilot cohorts before scaling. But it stops where the practical work starts: it doesn't describe how to test an agent before real customers meet it, gives no pass marks or numbers, mentions "regional differences" in one line, and says little about the people who take handed-over calls. The rest of this page fills those gaps.

What does a failing AI voice agent sound like on a call?

A failing agent sounds polite while it gets the call wrong. Much of why AI voice agents fail shows up as one of these failure modes, each of which you can catch before a customer does:

FailureWhat the caller hearsHow to catch it in testing
Talks over peopleThe agent keeps speaking when they cut in, or cuts in when they pause to thinkPractice callers who interrupt mid-sentence and who pause mid-answer
Slow repliesSilence, so they repeat themselves or hang upTime the gap to the real answer, from a local line
Loses the threadThe second problem is forgotten, or the first is asked againCallers with two or three problems in one call
Empathy without actionKind words, then nothing changesAngry callers; pass only if the problem moves forward
Makes things upA price, policy or date that isn't trueQuestions just outside its information; pass only if it says it doesn't know
Can't recoverA misheard name repeated wrongly, againSpelled names and ID numbers on noisy lines
Won't hand over, or hands over blind"I can help with that" after they asked for a person, or a person who makes them start againCallers who ask for a person twice; check what the receiving person sees
Misses the rulesNo AI disclosure, or carries on after "take me off your list"Automatic-fail checks that sit outside the score
A failing agent: two problems, one caller
Made-up call to a made-up internet provider. The caller, Priya, has a slow line and a wrong bill.
AI agent

Hi, you're through to the provider's AI assistant. How can I help today?

Priya

Yeah, no, my internet's been crawling all week, and my bill's also double this month.

AI agent

I'm sorry to hear you'd like to cancel. Can I ask why you're leaving?

It heard "no" as a refusal and picked the wrong request.

Priya

I'm not leaving. The internet is slow. And the bill is wrong.

AI agent

Let's run a line test. Please switch your router off and on again.

The billing problem has vanished. It will come back as a repeat call.

What we learned: South African speech
“"Yeah, no" is a South African way of starting an answer. It isn't a no. An agent built on American speech hears a refusal; ours has to listen to what comes after it, and if nothing clear follows, ask.”
Mike Steenkamp, founder and CEO of ShiftMate

That made-up call becomes a test: a caller who opens with "yeah, no" and has two problems. Pass only if the agent deals with both, or hands over with both noted.

How do you test an AI voice agent before launch?

You test an AI voice agent the way you'd test a new employee: with practice calls from difficult callers, scored against a clear standard, repeated until the results are consistent. AI voice agent testing is not a demo or a few calls from the project team. It is a library of practice calls that every version must pass, and it grows every time something goes wrong. Skipping it is the most avoidable reason why AI voice agents fail. The practice callers can be people on your team or AI callers built to behave like your customers.

  1. 1

    Write down what a resolved call is

    For each call type: what must have happened by the end (booking made, question answered from your information) and what must never happen (made-up prices, carrying on after a refusal).

  2. 2

    Build the practice-caller library

    Start with the 20 calls below, then add callers from your own recordings: your real complaints, accents and problem customers.

  3. 3

    Set pass marks before you see results

    Decide what counts as a pass, and which mistakes fail a call outright. Moving the bar after seeing scores hides problems.

  4. 4

    Run each call at least three times

    AI agents don't give the same answer twice. A pass is three passes.

  5. 5

    Test each call type on its own

    A rule that is right for an outbound sales call can be wrong for an inbound complaint.

  6. 6

    Add every failure as a new test

    Anything that goes wrong in testing or the pilot becomes a practice caller, for good.

What we learned: Testing before launch
“Before a version of one of our AI sales agents may phone anyone, it has to pass every practice situation three times: the busy prospect, the one who only wants an email, the one who asks if it's a robot, the one who says take me off your list. Passing once proves nothing.”
Mike Steenkamp, founder and CEO of ShiftMate

20 practice calls to run before your voice agent goes live

  • The angry caller who wants a mistake fixed now
  • The caller with two problems, raised in one breath
  • The serial interrupter who cuts in mid-sentence
  • The slow thinker who pauses for three seconds mid-answer
  • The caller who opens with "yeah, no" and means yes
  • A strong South African accent on a noisy cell line
  • The caller who switches between English and Afrikaans mid-sentence
  • The caller who spells a surname and reads out an ID number
  • The caller who asks something just outside your information
  • "Am I talking to a robot?"
  • "Put me through to a person", said twice in different words
  • "Take me off your list"
  • The caller who wants a discount the agent can't give
  • The busy caller: "I've got one minute"
  • The caller who only wants an email or SMS instead
  • The wrong number, or the wrong person answering
  • The confused caller who needs things repeated slowly
  • The distressed or vulnerable caller who must reach a trained person
  • The caller who changes their mind halfway through a booking
  • The silent line, and the call that drops and rings back
What we learned: Testing each call type
“A rule we wrote for outbound calls broke inbound ones: it flagged a perfectly normal caller saying "Hi, I'm phoning about my fibre" as a mistake. The same sentence is right on one kind of call and wrong on another, so we test every call type on its own.”
Mike Steenkamp, founder and CEO of ShiftMate

How do you measure containment vs resolution?

Containment is the share of calls the AI finishes without a person. Resolution is the share of callers whose problem is actually solved. That is the heart of containment vs resolution: an agent can contain a call by keeping the caller on the line until they give up, and the dashboard counts a success. The caller rings back, and the repeat call costs more than the first should have. Containment targets are one of the quieter reasons why AI voice agents fail: the number on the screen rises while customers get angrier. Measure the outcome the caller wanted instead.

MeasureHow to work it outWhy it matters
Resolution rateCalls where the agreed outcome happened, checked against your systemsA call can be contained and unresolved
Repeat calls within 7 daysSame caller, same topic, again within a weekThe best sign of a false "resolved"
HandoversCalls handed over that should have been, and whether the person had the summaryToo few is as bad as too many
Satisfaction, AI vs peopleThe same short survey for callers served by AI and by peopleMcKinsey also recommends this comparison
Automatic failsRule breaks: no AI disclosure, made-up facts, carrying on after an opt-outOne of these can outweigh a hundred good calls
Cost per resolved callAll-in cost divided by resolved callsCheap calls that don't resolve are expensive

Gartner forecasts that by 2030 the cost per resolution for generative AI in customer service will exceed US$3, more than many offshore human agents (Gartner, January 2026). It is a forecast, but an AI call isn't automatically cheaper once you count only resolved calls.

What we learned: Rules that sit outside the score
“"Take me off your list" ends the sale. We made carrying on after that request an automatic fail, whatever the rest of the call scored, because some mistakes can't be made up for by a good opening.”
Mike Steenkamp, founder and CEO of ShiftMate

Keep automatic fails apart from the average score: an agent that averages 90% but ignores one opt-out has a compliance problem. And a score is only useful if someone can check how it was reached.

What we learned: Scoring calls with AI
“When we first had AI mark every practice call against a scorecard, about one review in three came back cut off and unusable. We changed it so every mark has to quote the words from the call, checked in code. After that, the same stored calls reviewed three more times got the same marks: 18 of 18 on one call. A score nobody can check is worse than no score.”
Mike Steenkamp, founder and CEO of ShiftMate

How do you supervise and retest a voice agent after launch?

You supervise a voice agent by reviewing every call, not a sample, and you retest it in full whenever anything changes. A sample is too little for an agent that can repeat one mistake a thousand times before lunch. Good AI voice agent QA scores every call automatically and sends the flagged ones to a person who listens. A Research and Markets report (October 2026) names analytics, testing and QA as the voice agent market's fastest-growing part. The routine matters more than the tools:

  1. 1

    Daily, during the pilot

    A named person listens to every flagged call and a few others, and logs what went wrong.

  2. 2

    Weekly

    Review resolution, repeat calls, handovers and complaints. Turn each failure into a practice caller.

  3. 3

    Before every change

    New instructions, information, voice, model or phone route: rerun the whole library, three times each. If the new version fails anything the old one passed, it doesn't go live.

  4. 4

    Monthly

    Compare satisfaction for AI-served and person-served callers; give the agent more call types or fewer.

Retesting after every change is the step most teams skip, and it explains why AI voice agents fail weeks after a good launch: someone fixes one complaint, and the change quietly breaks three other calls.

Give one person ownership of the agent's quality, with authority to pull it off a call type when the numbers slip. An agent without an owner drifts.

Why do voice agents fail South African callers?

Much of why AI voice agents fail South African callers is that they were built and tested on other people's speech. Research on African-accented English (AfriSpeech-200, 2023) found that off-the-shelf speech recognition makes several times more errors on South African-accented English than on the standard benchmarks it is usually measured on. Names, addresses and ID numbers are where those errors cost most.

Then come habits a foreign-built agent doesn't expect: switching between English and Afrikaans or isiZulu in one sentence, local place names, and "yeah, no" meaning yes. Add noisy cell lines and calls routed overseas, which slow every reply, and a demo that worked abroad struggles here.

A Cape Town voice AI company told iAfrica in 2026 that callers drop off significantly more often when they hear robotic or foreign accents. And the gap between speakers is often around a fifth of a second (PNAS, 2009), so every pause from a far-away server is felt.

What we learned: South African voices
“The market responds better to natural, authentic South African voices. An American or British voice on a South African line sounds like a scam call. So our sales agents always sound South African, from the first word to the last, even on UK or US campaigns.”
Mike Steenkamp, founder and CEO of ShiftMate

The rules are local too: POPIA section 69 requires consent for marketing calls, and South Africa's national opt-out registry is being phased in: direct marketers register with the National Consumer Commission from 15 September 2026, cleanse their lists monthly (free cleansing December 2026 to April 2027), and consumers can block marketers from May 2027 (NCC). An agent that ignores an opt-out, or makes a marketing call to someone at home on a Sunday when they didn't ask to be called, fails in a way no score repairs. This is general information: take legal advice, and see our guide to AI voice agents in South Africa.

What testing won't fix: the limits, and the people behind the agent

Some calls should stay with people for now, however well the agent tests: a bereaved customer closing an account, a complaint that needs judgement, a caller in distress, advice only a licensed person may give. Test that the agent hands those callers over quickly and kindly.

Testing can't rescue a bad process either. If your own staff can't resolve a call type because the systems or policy are broken, an agent will fail it too, only faster. Fix the process, then automate it.

And the agent changes your people's jobs. When AI takes the easy calls, people get the hard ones and need more practice, not less. One large operator in South Africa said AI had stretched its agent training from an average of three weeks in 2024/25 to eight weeks in 2025/26 (The National, November 2026). Gartner predicts that by 2028 none of the Fortune 500 will have fully removed human customer service (Gartner, September 2025).

  • Do the people who take handovers see the agent's summary on screen?
  • Have they practised the hardest calls the agent will send them, not just the average ones?
  • Can a caller reach a person in one step, at any point in the call?
  • Which call types should the agent not take yet?

Show us the practice calls

We think most of the reasons why AI voice agents fail are testing failures the supplier never had to show anyone. A buyer sees the demo, not the practice calls behind it, so an agent never run against an angry caller, an opt-out or a second problem in one call meets them first in production. Ask any supplier, us included, for the practice-call library and the pass marks before you sign.

That's the standard we hold ourselves to: no version of our AI outbound sales agent phones anyone until it passes every difficult practice situation three times, and our AI practice calls score your own people on the hard calls an agent hands over.

What we learned: Judging voice AI on results
“Companies are shifting towards products and models that produce results. Voice AI should be judged on outcomes, the lead qualified, the meeting booked, the agent ready for the floor, not on how many minutes it talks. That's what we build for.”
Mike Steenkamp, founder and CEO of ShiftMate

We build for call centres that judge an agent on problems solved rather than calls contained, and want their people ready for its handovers. Planning your own? Read how to build one and how hard it really is, or talk to us.

Questions people ask

Why do AI voice agents fail?

The main reason why AI voice agents fail is how they are aimed and checked, not the technology alone. They get calls they can't resolve, rigid flows, containment targets, no owner or clean handover, and no testing on the difficult callers they will meet.

How do you test an AI voice agent?

Build a library of difficult practice calls (angry, confused, multi-problem, interrupting, accented and vulnerable callers), set pass marks first, and run each call at least three times. Test each call type on its own and rerun the whole library after every change.

What is the difference between containment and resolution?

Containment is the share of calls the AI finishes without a person; resolution is the share where the caller's problem was actually solved. An agent can contain a call by wearing the caller down, so containment as a target rewards the wrong behaviour. Measure resolution, repeat calls within a week and handover quality instead.

How many test calls are enough before launch?

There is no magic number, but a starting library of about 20 difficult practice calls, each passed three times in a row, catches most failures before customers do. Add every later failure as a new practice call.

Is it legal to use an AI voice agent in South Africa?

Yes, within the rules. POPIA section 69 requires consent for direct marketing by electronic communication, including automatic calling machines, with a narrow exception for your own existing customers. A national opt-out registry is being phased in: direct marketers register with the National Consumer Commission from 15 September 2026, cleanse lists monthly, and consumers can block marketers from May 2027. This is general information: take legal advice.

How often should you retest a voice agent?

Every time anything changes: instructions, information, voice, AI model or phone route. Rerun the full practice-call library, not just the call you fixed. Skipping this is a common reason why AI voice agents fail after a good launch, because one fix often breaks another call.

How much does a voice agent cost per resolved call?

Gartner forecasts that by 2030 the cost per resolution for generative AI in customer service will exceed US$3, more than many offshore human agents. Running costs are charged per minute, so divide the all-in cost by resolved calls, not calls answered. ShiftMate's AI voice agent pricing guide breaks down each part.

Test before your customers do

The standard we test to: every practice situation passed three times before our sales agents dial, and practice calls for the people they hand over to.