Building voice agents

How hard is it to build a voice agent? Easy to demo, hard to trust

The short answer

How hard is it to build a voice agent? A demo is easy: a developer can make one in days. One you'd trust with customers is hard. The work is in fast replies, natural turn-taking, accents and noisy lines, staying on script, never making things up, clean handovers, and proving all of that again on every new version.

By , founder and CEO of ShiftMateUpdated 14 min read
Testing a voice agent on a practice call with a difficult AI customer
Mike Steenkamp, founder and CEO of ShiftMate, a South African voice agent specialist
“We build voice agents for a living at ShiftMate in South Africa. Getting one to sound right is the easy part; getting it to reply fast, understand local callers and never promise what the system hasn't done is the hard part. This guide explains why, and what it really takes.”
Mike Steenkamp, founder and CEO of ShiftMate, Umhlanga, South Africa
Key takeaways
  • How hard is it to build a voice agent? A demo takes days; an agent you trust with every caller takes months, and most of that is testing. Our complete guide to AI voice agents covers the basics.
  • Speed is the first wall: gaps between speakers in normal conversation are often around a fifth of a second, and callers notice every extra pause.
  • The hardest work is proving reliability: every version must pass every difficult practice call, several times, because a fix in one place can break another.
  • South African callers add accents, language-switching and noisy lines: off-the-shelf speech recognition makes several times more errors on African-accented English.
  • Build it yourself only if voice is your product and you can fund months of testing. Otherwise use a platform or have it built.

How hard is it to build a voice agent?

Building a voice agent that works in a demo is easy; building one you would trust with your customers is hard. The parts are available to anyone, and a developer can connect a phone line, speech recognition, an AI model and a synthetic voice in days. That demo will impress your team, because your team is friendly, speaks clearly in a quiet room and asks the questions it was built for.

So when people ask "How hard is it to build a voice agent?", the real question is how hard it is to make one reliable. Real callers interrupt, mumble, change their minds, raise two problems at once, call on bad lines and ask things nobody planned for. Each of those needs a rule, and each rule needs a test that proves the agent follows it every time. That is where the months go.

It shows in the market. In a Gartner survey of 187 customer service leaders in mid-2024, 44% were exploring a customer-facing voicebot and 11% were piloting one, but only 5% had one deployed. Plenty of teams get a demo working; far fewer get to an agent they let loose on every call.

GoalHow hardWhy
A demo that impresses your teamEasy: daysFriendly testers, quiet rooms, happy-path questions
A pilot on a small share of real callsMedium: weeksReal callers interrupt, mumble and go off topic; people must review every call
An agent you trust with every callerHard: monthsIt must be right every time, on every version, and you must be able to prove it

What are the hardest voice agent challenges?

Ask anyone who has done it, "How hard is it to build a voice agent?", and you will hear the same list. These eight voice agent challenges take most of the time, and most AI voice agent problems that reach customers trace back to one of them:

1

Speed

The agent must listen, understand, decide and speak inside a gap people accept. Long pauses make callers talk over it, repeat themselves or hang up.

2

Turn-taking

Knowing when the caller has finished rather than paused, stopping mid-sentence when they cut in, and not stopping for a cough or a word to someone else in the room.

3

Accents, noise and bad lines

Speech recognition makes far more mistakes on some accents, on cell lines and in noisy places. Names, numbers and addresses suffer most.

4

Staying on script

AI models are eager to help. Without firm rules they promise discounts, invent policies and answer questions they shouldn't.

5

Never making things up

The agent must answer only from your information, or say it doesn't know, and you must be able to prove it does.

6

Doing, not just saying

Bookings, updates and messages must really happen in your systems before the agent tells the caller they are done.

7

Clean handovers

Spotting when a person is needed and passing the call on with a summary, so the caller never starts again.

8

Proving it on every version

Every change can fix one call and break another. Without a full retest, customers find the break first.

None of these is a single setting. Each is a trade-off: an agent tuned to reply faster interrupts more; one told to be careful hands over too often; one told to be helpful starts promising things. Getting the balance right for your calls is the real work.

Why is voice AI latency so hard to get right?

Voice AI latency is hard because people are very fast at conversation and every part of a voice agent adds delay. Research across ten languages found the gap between one person finishing and the next starting is often around a fifth of a second (Stivers and others, PNAS 2009). A voice agent has to decide the caller has finished, understand them, write a reply, start the voice and send the audio down a phone line, and each step takes time.

Worse, the delay you measure in a lab is not the delay a caller feels. A server near the AI gives a flattering number; a caller on a cell line in Durban, with the call routed overseas and back, hears something slower. If your question is "How hard is it to build a voice agent that feels natural?", this is the first wall you hit.

What we learned: Speed (latency)
“We ran hundreds of test calls to get the delay right. A voice agent can say all the right things and still fail if the pause before each reply is too long: callers talk over it, repeat themselves or hang up. Speed isn't a setting you switch on; it's something you measure on every version, from where your callers really are.”
Mike Steenkamp, founder and CEO of ShiftMate

Speed also fights with quality. A bigger, more careful AI model often gives better answers but takes longer to start speaking. A faster one may be good enough for simple calls and not for hard ones. You only find the balance by testing real call types, end to end.

What we learned: Speed (latency)
“One of our earlier AI interviewers was good, but slow to reply, and on its first test run it dropped a turn. That taught us that a clever answer delivered late loses to a simple answer delivered on time, so we rebuilt for speed.”
Mike Steenkamp, founder and CEO of ShiftMate

There is a tempting shortcut: start the reply with a quick filler word so the pause disappears. It makes the speed number look better without making the call better.

What we learned: Speed (latency)
“We learned to measure the real answer, not the filler. A quick "Sure" makes a speed number look good while the caller is still waiting for what they asked. And we measure the round trip from a South African caller, not from a server next door to the AI.”
Mike Steenkamp, founder and CEO of ShiftMate

How to measure voice AI latency honestly

  • Measure from the moment the caller stops speaking to the first word of the real answer, not a filler
  • Make test calls on real phone lines from where your callers are, including cell lines
  • Measure on every version, because a new instruction, model or voice can change it
  • Look at the slowest calls, not the average: callers remember the long pause
  • Listen for the side effects: talking over the caller, or stopping for background noise

Why does testing take most of the time?

Testing takes most of the time because a voice agent doesn't give the same answer twice, and because every fix can break something else. The honest answer to "How hard is it to build a voice agent?" is mostly an answer about testing: the happy path takes an afternoon; proving the other calls takes months.

You need a library of difficult practice calls (the angry caller, the interrupter, the caller with two problems, the one who asks for a person, the one who says take me off your list) and you need to run every one of them, several times, on every new version. Each failure you find becomes a new test. The library only grows.

What you testWhy it's hardWhat good looks like
Each call type on its ownA rule that's right for one call type can be wrong for anotherSeparate pass marks for inbound, outbound and reminder calls
The same call, several timesAI agents vary from run to runA pass is three passes in a row
Every versionFixes break other callsThe full library reruns before any change goes live
The scoring itselfAI marking can be cut off, vague or inconsistentEvery mark quotes the words from the call
The practice callersPretend customers can give in too easilyChecked against real call recordings
What we learned: Testing each call type
“A rule we wrote for outbound calls broke inbound ones: it flagged a perfectly normal caller saying "Hi, I'm phoning about my fibre" as a mistake. The same sentence is right on one kind of call and wrong on another, so we test every call type on its own.”
Mike Steenkamp, founder and CEO of ShiftMate
What we learned: Scoring calls with AI
“When we first had AI mark every practice call against a scorecard, about one review in three came back cut off and unusable. We changed it so every mark has to quote the words from the call, checked in code. After that, the same stored calls reviewed three more times got the same marks: 18 of 18 on one call. A score nobody can check is worse than no score.”
Mike Steenkamp, founder and CEO of ShiftMate

For a full test plan with 20 practice calls and the numbers to track, see why AI voice agents fail.

Is it harder to build a voice agent for South African callers?

Yes. Research on African-accented English (AfriSpeech-200, 2023) found that off-the-shelf speech recognition makes several times more errors on these accents, including South African ones, than on the standard benchmarks it is usually measured on. AI voices and models are often trained mostly on American and British speech, so local names, places and habits trip them up.

Callers switch between English and Afrikaans or isiZulu in a sentence, many calls come in on noisy cell lines, and some everyday phrases mean the opposite of what a foreign-built agent expects. So how hard is it to build a voice agent for South Africa? Harder than the demo suggests, and impossible to judge without testing on your own callers.

What we learned: South African speech
“"Yeah, no" is a South African way of starting an answer. It isn't a no. An agent built on American speech hears a refusal; ours has to listen to what comes after it, and if nothing clear follows, ask.”
Mike Steenkamp, founder and CEO of ShiftMate

The voice matters as much as the ears. A Cape Town voice AI company told iAfrica in 2026 that callers drop off significantly more often when they hear robotic or foreign accents.

What we learned: South African voices
“The market responds better to natural, authentic South African voices. An American or British voice on a South African line sounds like a scam call. So our sales agents always sound South African, from the first word to the last, even on UK or US campaigns.”
Mike Steenkamp, founder and CEO of ShiftMate

And the rules are local. POPIA section 69 requires consent for direct marketing by electronic communication, including automatic calling machines, with a narrow exception for your own existing customers. South Africa's national opt-out registry is being phased in: direct marketers register with the National Consumer Commission from 15 September 2026, cleanse their lists monthly (free cleansing December 2026 to April 2027), and consumers can block marketers from May 2027 (NCC). Building those rules in is part of the job. This is general information: take legal advice for your own calls.

What does building a voice agent cost in time and effort?

Building a voice agent costs far more in testing and supervision than in the first build. When you budget, count the work after the demo: the useful question is not just "How hard is it to build a voice agent?" but how hard it is to keep one working well a year from now.

StageTypical timeThe work
DemoDaysConnect the parts, write a first set of instructions, try a few friendly calls
Writing the callDays to weeksStudy real recordings, define a resolved call, the rules, the handover
Connecting systemsWeeksBooking, customer records, order lookup, with safe limits on what the agent may do
TestingWeeks to monthsBuild and run the practice-call library, fix, rerun, again and again
PilotWeeksA small share of real calls, every flagged call reviewed daily
Running itEvery week, for goodReview calls, fix failures, retest each change, watch the numbers
  • People: someone who understands real-time audio and phone systems, someone who writes the call, and an owner who reviews calls weekly
  • Running costs per minute: phone line, speech recognition, AI model and voice, plus hosting
  • Testing time: hundreds of practice calls, repeated on every change
  • Compliance: consent, recording notices, privacy and, in some industries, the regulator's rules
  • Training: the people who take the handed-over calls need practice for the harder calls left to them

For what each part costs per minute, see our AI voice agent pricing guide. For the build steps in order, see how to build an AI voice agent.

When should you build a voice agent yourself, and when not to?

Build it yourself when voice is the heart of your product and you can fund months of testing; otherwise use a platform or have it built. Whatever you choose, a voice agent won't fix a broken process, won't replace judgement on sensitive calls, and won't train your people for the harder calls it leaves them.

Build it yourself if...Use a platform or have it built if...
Voice is the heart of your productVoice is a channel for your business
You have engineers with real-time audio and phone experienceYour team's time is better spent on your customers
You need control of every part, including where data is keptYou need results in weeks, not months
You can fund months of testing and weekly reviews for goodYou want someone accountable for quality

And don't build one at all, yet, if your own staff can't resolve the call type, if the calls need comfort or licensed advice, if your call volumes are too small to repay the testing, or if nobody can own the agent after launch. In the end, "How hard is it to build a voice agent?" is really a question about which of these camps you are in.

The hard part, hired rather than built

We think the honest answer to "How hard is it to build a voice agent?" explains why voice AI has belonged to large companies with engineering teams. That's changing. The hard months, the latency work, the accent testing and the retest after every change, are work a specialist can carry once and use for many businesses. We see that as the real shift: the hard part becomes something you can hire rather than something you must build.

We have worked through these problems in our own products: our AI sales agents must clear every difficult practice situation three times before they're allowed to dial, our AI outbound sales agent is built on that rule, and our AI practice calls train the people who take the calls an agent hands over.

If you've asked "How hard is it to build a voice agent?" and decided you'd rather not, talk to us.

What we learned: Who we want to work with
“We're open about who we want to work with: call centres and sales teams that want their people trained, their leads worked and the low-value calls taken off their operators, and businesses that could never afford a call centre but still need their phones answered and their leads called.”
Mike Steenkamp, founder and CEO of ShiftMate

Questions people ask

How hard is it to build a voice agent?

Easy to demo, hard to make reliable. A developer can build a demo in days; an agent you trust with real customers needs fast replies, good turn-taking, accent handling, strict rules and extensive testing on every version, which usually takes months.

What is the hardest part of building a voice agent?

Proving it is reliable: staying on script, never making things up, handling interruptions and accents, and showing all of that again on every new version. A fix in one place often breaks another call, so the whole practice-call library must be rerun before each change.

Why does voice AI latency matter so much?

Because people are fast at conversation: the gap between speakers is often around a fifth of a second. Long pauses make callers talk over the agent, repeat themselves or hang up. Measure the delay to the real answer, from where your callers are.

What are the most common AI voice agent problems and mistakes?

Slow replies, talking over callers, mishearing names and numbers, making up answers, promising things it can't do, losing track of a second problem, refusing to hand over, and handing over without a summary. Most can be caught with difficult practice calls before launch.

Do voice agents understand South African accents?

It varies a lot, so test before you trust. Research on African-accented English found off-the-shelf speech recognition makes several times more errors than on standard benchmarks. Test with your own callers' accents, language-switching and noisy cell lines.

How long does it take to build a voice agent?

A demo takes days. A pilot on a small share of real calls usually takes weeks of writing, connecting and testing. An agent you trust with every caller takes months, and after launch it needs weekly reviews and a full retest after every change.

How much does it cost to build a voice agent?

Running costs are charged per minute, covering the phone line, speech recognition, the AI model, the voice and hosting. Most of the real cost is people's time: writing the call, connecting systems, testing hundreds of practice calls and reviewing calls every week. ShiftMate's AI voice agent pricing guide breaks down the per-minute parts.

Is it legal to use an AI voice agent in South Africa?

Yes, within the rules. POPIA section 69 requires consent for direct marketing by electronic communication, including automatic calling machines, with a narrow exception for your own existing customers. A national opt-out registry is being phased in: direct marketers register with the National Consumer Commission from 15 September 2026, cleanse lists monthly, and consumers can block marketers from May 2027. Marketing calls to consumers at home keep to set hours unless they asked to be called. This is general information: take legal advice.

Skip the hard months

We carry the testing and tuning; you get an agent that has already met your difficult callers in practice.