How hard is it to build a voice agent? A demo is easy: a developer can make one in days. One you'd trust with customers is hard. The work is in fast replies, natural turn-taking, accents and noisy lines, staying on script, never making things up, clean handovers, and proving all of that again on every new version.


“We build voice agents for a living at ShiftMate in South Africa. Getting one to sound right is the easy part; getting it to reply fast, understand local callers and never promise what the system hasn't done is the hard part. This guide explains why, and what it really takes.”
Building a voice agent that works in a demo is easy; building one you would trust with your customers is hard. The parts are available to anyone, and a developer can connect a phone line, speech recognition, an AI model and a synthetic voice in days. That demo will impress your team, because your team is friendly, speaks clearly in a quiet room and asks the questions it was built for.
So when people ask "How hard is it to build a voice agent?", the real question is how hard it is to make one reliable. Real callers interrupt, mumble, change their minds, raise two problems at once, call on bad lines and ask things nobody planned for. Each of those needs a rule, and each rule needs a test that proves the agent follows it every time. That is where the months go.
It shows in the market. In a Gartner survey of 187 customer service leaders in mid-2024, 44% were exploring a customer-facing voicebot and 11% were piloting one, but only 5% had one deployed. Plenty of teams get a demo working; far fewer get to an agent they let loose on every call.
| Goal | How hard | Why |
|---|---|---|
| A demo that impresses your team | Easy: days | Friendly testers, quiet rooms, happy-path questions |
| A pilot on a small share of real calls | Medium: weeks | Real callers interrupt, mumble and go off topic; people must review every call |
| An agent you trust with every caller | Hard: months | It must be right every time, on every version, and you must be able to prove it |
Ask anyone who has done it, "How hard is it to build a voice agent?", and you will hear the same list. These eight voice agent challenges take most of the time, and most AI voice agent problems that reach customers trace back to one of them:
The agent must listen, understand, decide and speak inside a gap people accept. Long pauses make callers talk over it, repeat themselves or hang up.
Knowing when the caller has finished rather than paused, stopping mid-sentence when they cut in, and not stopping for a cough or a word to someone else in the room.
Speech recognition makes far more mistakes on some accents, on cell lines and in noisy places. Names, numbers and addresses suffer most.
AI models are eager to help. Without firm rules they promise discounts, invent policies and answer questions they shouldn't.
The agent must answer only from your information, or say it doesn't know, and you must be able to prove it does.
Bookings, updates and messages must really happen in your systems before the agent tells the caller they are done.
Spotting when a person is needed and passing the call on with a summary, so the caller never starts again.
Every change can fix one call and break another. Without a full retest, customers find the break first.
None of these is a single setting. Each is a trade-off: an agent tuned to reply faster interrupts more; one told to be careful hands over too often; one told to be helpful starts promising things. Getting the balance right for your calls is the real work.
Voice AI latency is hard because people are very fast at conversation and every part of a voice agent adds delay. Research across ten languages found the gap between one person finishing and the next starting is often around a fifth of a second (Stivers and others, PNAS 2009). A voice agent has to decide the caller has finished, understand them, write a reply, start the voice and send the audio down a phone line, and each step takes time.
Worse, the delay you measure in a lab is not the delay a caller feels. A server near the AI gives a flattering number; a caller on a cell line in Durban, with the call routed overseas and back, hears something slower. If your question is "How hard is it to build a voice agent that feels natural?", this is the first wall you hit.
“We ran hundreds of test calls to get the delay right. A voice agent can say all the right things and still fail if the pause before each reply is too long: callers talk over it, repeat themselves or hang up. Speed isn't a setting you switch on; it's something you measure on every version, from where your callers really are.”
Speed also fights with quality. A bigger, more careful AI model often gives better answers but takes longer to start speaking. A faster one may be good enough for simple calls and not for hard ones. You only find the balance by testing real call types, end to end.
“One of our earlier AI interviewers was good, but slow to reply, and on its first test run it dropped a turn. That taught us that a clever answer delivered late loses to a simple answer delivered on time, so we rebuilt for speed.”
There is a tempting shortcut: start the reply with a quick filler word so the pause disappears. It makes the speed number look better without making the call better.
“We learned to measure the real answer, not the filler. A quick "Sure" makes a speed number look good while the caller is still waiting for what they asked. And we measure the round trip from a South African caller, not from a server next door to the AI.”
Testing takes most of the time because a voice agent doesn't give the same answer twice, and because every fix can break something else. The honest answer to "How hard is it to build a voice agent?" is mostly an answer about testing: the happy path takes an afternoon; proving the other calls takes months.
You need a library of difficult practice calls (the angry caller, the interrupter, the caller with two problems, the one who asks for a person, the one who says take me off your list) and you need to run every one of them, several times, on every new version. Each failure you find becomes a new test. The library only grows.
| What you test | Why it's hard | What good looks like |
|---|---|---|
| Each call type on its own | A rule that's right for one call type can be wrong for another | Separate pass marks for inbound, outbound and reminder calls |
| The same call, several times | AI agents vary from run to run | A pass is three passes in a row |
| Every version | Fixes break other calls | The full library reruns before any change goes live |
| The scoring itself | AI marking can be cut off, vague or inconsistent | Every mark quotes the words from the call |
| The practice callers | Pretend customers can give in too easily | Checked against real call recordings |
“A rule we wrote for outbound calls broke inbound ones: it flagged a perfectly normal caller saying "Hi, I'm phoning about my fibre" as a mistake. The same sentence is right on one kind of call and wrong on another, so we test every call type on its own.”
“When we first had AI mark every practice call against a scorecard, about one review in three came back cut off and unusable. We changed it so every mark has to quote the words from the call, checked in code. After that, the same stored calls reviewed three more times got the same marks: 18 of 18 on one call. A score nobody can check is worse than no score.”
For a full test plan with 20 practice calls and the numbers to track, see why AI voice agents fail.
Yes. Research on African-accented English (AfriSpeech-200, 2023) found that off-the-shelf speech recognition makes several times more errors on these accents, including South African ones, than on the standard benchmarks it is usually measured on. AI voices and models are often trained mostly on American and British speech, so local names, places and habits trip them up.
Callers switch between English and Afrikaans or isiZulu in a sentence, many calls come in on noisy cell lines, and some everyday phrases mean the opposite of what a foreign-built agent expects. So how hard is it to build a voice agent for South Africa? Harder than the demo suggests, and impossible to judge without testing on your own callers.
“"Yeah, no" is a South African way of starting an answer. It isn't a no. An agent built on American speech hears a refusal; ours has to listen to what comes after it, and if nothing clear follows, ask.”
The voice matters as much as the ears. A Cape Town voice AI company told iAfrica in 2026 that callers drop off significantly more often when they hear robotic or foreign accents.
“The market responds better to natural, authentic South African voices. An American or British voice on a South African line sounds like a scam call. So our sales agents always sound South African, from the first word to the last, even on UK or US campaigns.”
And the rules are local. POPIA section 69 requires consent for direct marketing by electronic communication, including automatic calling machines, with a narrow exception for your own existing customers. South Africa's national opt-out registry is being phased in: direct marketers register with the National Consumer Commission from 15 September 2026, cleanse their lists monthly (free cleansing December 2026 to April 2027), and consumers can block marketers from May 2027 (NCC). Building those rules in is part of the job. This is general information: take legal advice for your own calls.
Building a voice agent costs far more in testing and supervision than in the first build. When you budget, count the work after the demo: the useful question is not just "How hard is it to build a voice agent?" but how hard it is to keep one working well a year from now.
| Stage | Typical time | The work |
|---|---|---|
| Demo | Days | Connect the parts, write a first set of instructions, try a few friendly calls |
| Writing the call | Days to weeks | Study real recordings, define a resolved call, the rules, the handover |
| Connecting systems | Weeks | Booking, customer records, order lookup, with safe limits on what the agent may do |
| Testing | Weeks to months | Build and run the practice-call library, fix, rerun, again and again |
| Pilot | Weeks | A small share of real calls, every flagged call reviewed daily |
| Running it | Every week, for good | Review calls, fix failures, retest each change, watch the numbers |
For what each part costs per minute, see our AI voice agent pricing guide. For the build steps in order, see how to build an AI voice agent.
Build it yourself when voice is the heart of your product and you can fund months of testing; otherwise use a platform or have it built. Whatever you choose, a voice agent won't fix a broken process, won't replace judgement on sensitive calls, and won't train your people for the harder calls it leaves them.
| Build it yourself if... | Use a platform or have it built if... |
|---|---|
| Voice is the heart of your product | Voice is a channel for your business |
| You have engineers with real-time audio and phone experience | Your team's time is better spent on your customers |
| You need control of every part, including where data is kept | You need results in weeks, not months |
| You can fund months of testing and weekly reviews for good | You want someone accountable for quality |
And don't build one at all, yet, if your own staff can't resolve the call type, if the calls need comfort or licensed advice, if your call volumes are too small to repay the testing, or if nobody can own the agent after launch. In the end, "How hard is it to build a voice agent?" is really a question about which of these camps you are in.
We think the honest answer to "How hard is it to build a voice agent?" explains why voice AI has belonged to large companies with engineering teams. That's changing. The hard months, the latency work, the accent testing and the retest after every change, are work a specialist can carry once and use for many businesses. We see that as the real shift: the hard part becomes something you can hire rather than something you must build.
We have worked through these problems in our own products: our AI sales agents must clear every difficult practice situation three times before they're allowed to dial, our AI outbound sales agent is built on that rule, and our AI practice calls train the people who take the calls an agent hands over.
If you've asked "How hard is it to build a voice agent?" and decided you'd rather not, talk to us.
“We're open about who we want to work with: call centres and sales teams that want their people trained, their leads worked and the low-value calls taken off their operators, and businesses that could never afford a call centre but still need their phones answered and their leads called.”
Easy to demo, hard to make reliable. A developer can build a demo in days; an agent you trust with real customers needs fast replies, good turn-taking, accent handling, strict rules and extensive testing on every version, which usually takes months.
Proving it is reliable: staying on script, never making things up, handling interruptions and accents, and showing all of that again on every new version. A fix in one place often breaks another call, so the whole practice-call library must be rerun before each change.
Because people are fast at conversation: the gap between speakers is often around a fifth of a second. Long pauses make callers talk over the agent, repeat themselves or hang up. Measure the delay to the real answer, from where your callers are.
Slow replies, talking over callers, mishearing names and numbers, making up answers, promising things it can't do, losing track of a second problem, refusing to hand over, and handing over without a summary. Most can be caught with difficult practice calls before launch.
It varies a lot, so test before you trust. Research on African-accented English found off-the-shelf speech recognition makes several times more errors than on standard benchmarks. Test with your own callers' accents, language-switching and noisy cell lines.
A demo takes days. A pilot on a small share of real calls usually takes weeks of writing, connecting and testing. An agent you trust with every caller takes months, and after launch it needs weekly reviews and a full retest after every change.
Running costs are charged per minute, covering the phone line, speech recognition, the AI model, the voice and hosting. Most of the real cost is people's time: writing the call, connecting systems, testing hundreds of practice calls and reviewing calls every week. ShiftMate's AI voice agent pricing guide breaks down the per-minute parts.
Yes, within the rules. POPIA section 69 requires consent for direct marketing by electronic communication, including automatic calling machines, with a narrow exception for your own existing customers. A national opt-out registry is being phased in: direct marketers register with the National Consumer Commission from 15 September 2026, cleanse lists monthly, and consumers can block marketers from May 2027. Marketing calls to consumers at home keep to set hours unless they asked to be called. This is general information: take legal advice.
We carry the testing and tuning; you get an agent that has already met your difficult callers in practice.