How to build an AI voice agent, in short: connect a phone line to speech recognition, an AI model and a synthetic voice (or one speech-to-speech model), give it one clear job, your information and tools, then test it on hundreds of difficult practice calls before a small live pilot. A demo takes days; a reliable agent takes months.


“ShiftMate is a South African voice agent specialist, and we've built the parts this guide describes ourselves, including hundreds of test calls to get the delay right. Here's how to build an AI voice agent: the parts, the two main ways to put them together, the steps, and when building isn't worth it.”
An AI voice agent is software that holds a phone conversation: it listens, works out what the caller wants, does something useful (books, looks up, records, hands over) and replies in a natural voice, fast enough that the conversation feels normal. Learning how to build an AI voice agent means learning to make those pieces work together in real time, on real phone lines, with real callers who don't follow your script.
The good news: the parts are widely available, and a developer can make a working demo in days. The harder news: a demo isn't an agent you can trust with customers. Most of the work is in the calls that go wrong, which is why this guide spends as much time on testing and launch as on the build itself. If you want the honest view of the effort first, read how hard it is to build a voice agent.
It explains why so many projects stall. In a Gartner survey of 187 customer service leaders in mid-2024, 44% were exploring a voicebot and 11% piloting one, but only 5% had one deployed. The gap between those numbers is mostly testing, connections to real systems and trust, not the AI.
The first lesson in how to build an AI voice agent is that every agent has the same eight parts, whether you assemble them yourself or a platform hides them behind one screen. Knowing them tells you where problems will come from.
| Part | What it does | What goes wrong |
|---|---|---|
| Phone line or web call | Carries the audio: a phone number, a connection to your existing phone system, or a browser call | Poor audio, delay on long routes, recording consent |
| Speech recognition | Turns the caller's words into text | Accents, noise, names, numbers and addresses |
| Turn-taking | Decides when the caller has finished, and stops the agent when they cut in | Talking over people, or awkward silences |
| AI language model | Understands the request and decides what to say and do | Making things up, going off script, slow replies |
| Knowledge and tools | Your information, plus actions such as booking, looking up an order or sending an SMS | Outdated facts; actions it shouldn't take |
| Synthetic voice | Turns the reply into speech | Robotic delivery, wrong pronunciation of local names |
| Guardrails | Checks what the agent will say and do against your rules | Unapproved promises, prices and advice |
| Monitoring | Recordings, transcripts, outcomes, scores and alerts | Problems found by customers instead of by you |
Two parts are often forgotten in early builds and cause the most trouble later: turn-taking and guardrails. Turn-taking decides whether the call feels like a conversation or a walkie-talkie. Guardrails decide whether the agent can be trusted to speak for your business. Plan both from day one.
There are two main ways to put the parts together, and the choice of voice AI architecture is the biggest early decision in how to build an AI voice agent: it shapes speed, control and cost. A pipeline runs three steps one after another: speech recognition turns the caller's words into text, an AI model writes a text reply, and a synthetic voice speaks it. A speech-to-speech model listens to the audio and replies in audio directly, in one step.
| Pipeline (speech to text, AI model, voice) | Speech-to-speech (one model) | |
|---|---|---|
| Speed | Each step adds delay, so it needs careful tuning | Usually faster, with more natural turn-taking |
| Control | You can read and check every reply before it is spoken | Harder to check each word before the caller hears it |
| Voice choice | Any voice, including local South African voices | Limited to the voices the model offers |
| Swapping parts | Change one part without rebuilding the rest | All or nothing |
| Tone and emotion | Lost in the text step unless you add it back | Can carry tone from the caller's voice |
| Best for | Strict scripts, regulated calls, local voices | Natural, fast, open conversations |
Many working systems mix the two: a fast model handles the conversation while a separate check reviews anything risky, or a pipeline handles regulated calls while a speech-to-speech model handles open chat. There is no single right answer. Choose by asking three questions: how strict must the script be, how fast must the reply be, and which voice will your callers trust?
People expect a reply quickly: in normal conversation the gap between one person finishing and the other starting is often around a fifth of a second (Stivers and others, PNAS 2009). An agent spends time deciding the caller has finished, understanding them, writing a reply, starting the voice, and sending audio across the phone network. Each slice looks small; together they decide whether the call feels natural. Measure the whole round trip, on real phone lines, from where your callers are.
“We learned to measure the real answer, not the filler. A quick "Sure" makes a speed number look good while the caller is still waiting for what they asked. And we measure the round trip from a South African caller, not from a server next door to the AI.”
Here is how to build an AI voice agent from idea to live calls. The order matters: the steps people skip (writing the call down, testing, the pilot) are the ones that decide whether customers trust it.
Book an appointment, follow up a lead, confirm a delivery, give a claim status. Not "answer all our calls". Write down what a successful call ends with.
Listen to recordings of the call type you're automating: how callers open, the questions they ask, the accents, the complaints, the ways calls go wrong. These become your test callers later.
The stages, the facts the agent may use, the questions it must ask, what it must never say, and exactly when it hands over to a person.
Pipeline, speech-to-speech or a mix, and a voice your callers will trust. Write an opening that says it's an AI and that the call is recorded.
Booking system, customer records, order lookup. Give it as little access as the job allows, and make it confirm actions before it tells the caller they're done.
Rules it can't break whatever the caller says, and a handover that passes a summary to the person, so the caller never starts again.
Hundreds of them: interruptions, accents, noise, two problems in one call, angry callers, people who ask for a person. Each call passed several times, not once.
A few percent of calls, with people reviewing recordings every day and a quick way to switch the agent off.
Track resolution, handovers, repeat calls and complaints, not containment (containment vs resolution explains why). Turn each failure into a new practice call and rerun the full set before every change goes live.
“Our agents say they're an AI and that the call is recorded. People forgive a machine that's honest with them; they don't forgive one that pretends.”
“A spoken promise isn't a completed action. Our agents only tell a caller something is done once the system confirms it is done, not because the conversation went well.”
You test a voice agent with a library of difficult practice calls, scored against pass marks you set before you see results, and rerun on every new version. Any honest guide to how to build an AI voice agent puts the effort here: the happy path takes an afternoon; the other calls take months.
| Practice caller | What it tests | Pass when the agent... |
|---|---|---|
| Interrupts mid-sentence | Turn-taking | Stops, listens, carries on without repeating itself |
| Two problems in one breath | Keeping track | Deals with both, or hands over with both noted |
| Strong local accent on a noisy line | Understanding | Reads names and numbers back correctly |
| Asks something outside its information | Making things up | Says it doesn't know, offers a person |
| "Am I talking to a robot?" | Honesty | Says plainly that it's an AI |
| "Put me through to a person" | Handover | Hands over at once, with a summary |
| "Take me off your list" | Opt-outs | Stops selling, confirms, ends politely |
| Angry or distressed | Care | Slows down, acknowledges, hands over to a trained person |
“Before a version of one of our AI sales agents may phone anyone, it has to pass every practice situation three times: the busy prospect, the one who only wants an email, the one who asks if it's a robot, the one who says take me off your list. Passing once proves nothing.”
Test every call type on its own: a rule that is right for an outbound call can be wrong for an inbound one. And keep every failed call as a permanent test, so a fix in one place can't quietly break another. Our guide to why AI voice agents fail has a full 20-call test plan and the numbers to track.
The last stage of how to build an AI voice agent is the launch: start small, watch closely, and grow only when the numbers hold. Start with a few percent of calls or one lead source, with a named owner who reviews flagged calls daily. Expand when resolution and complaints are steady for several weeks, not when the first week looks good.
South Africa adds three things to plan for. First, accents and languages: research on African-accented English (AfriSpeech-200, 2023) found off-the-shelf speech recognition makes several times more errors on these accents than on the standard benchmarks, so test with your own callers and with language-switching. Second, voice: callers trust a voice that sounds like them. Third, the law.
“The market responds better to natural, authentic South African voices. An American or British voice on a South African line sounds like a scam call. So our sales agents always sound South African, from the first word to the last, even on UK or US campaigns.”
For outbound calls, POPIA section 69 requires consent for direct marketing by electronic communication, including automatic calling machines, with a narrow exception for your own existing customers. South Africa's national opt-out registry is being phased in: direct marketers register with the National Consumer Commission from 15 September 2026, cleanse their lists monthly (free cleansing December 2026 to April 2027), and consumers can block marketers from May 2027 (NCC). Calls to consumers at home, unless they asked to be contacted, keep to set hours: no Sundays or public holidays, Saturdays 09:00 to 13:00, other days 08:00 to 20:00. In the US, the FCC ruled in February 2024 that AI voices count as "artificial" under the TCPA, so consent rules apply. This is general information: take legal advice for your own calls.
Budget for running costs too. Most platforms charge per minute, and the all-in figure includes the phone line, speech recognition, the AI model and the voice, plus set-up and the time spent testing. Our AI voice agent pricing guide breaks down what each part costs.
Before you decide how to build an AI voice agent, decide who should build it. There are three ways to get an agent, and the right one depends on whether voice is your product or a channel for your business.
| Build it yourself | Developer or no-code platform | Done for you | |
|---|---|---|---|
| You need | Engineers who know real-time audio, phone systems and AI | Someone technical to configure, connect and test | Your scripts, rules, information and time to review calls |
| Time to a demo | Weeks | Days | Days to weeks |
| Time to reliable | Months | Weeks to months | Weeks |
| Control | Full | Within the platform's limits | Agreed with the supplier |
| Who tests and tunes it | You | You | The supplier, with you |
| Best for | Voice is your product | Tech teams with time to iterate | Businesses that want the outcome, not a project |
Whichever route you choose, keep ownership of three things: the definition of a good call, the practice-call library, and the right to see every recording. A supplier who won't let you run your own test calls before you sign is telling you something.
Sometimes the best answer to how to build an AI voice agent is: don't, yet. Skip it, or wait, when:
A voice agent doesn't replace training your people either. When it takes the simple calls, the ones left for people are harder, and they need practice for them.
Our view on build or buy is simple. If voice is your product, build it. If voice is a channel for your business, we believe your team's time is better spent on the script, the rules and the people than on audio engineering. We think more businesses will buy the result rather than build the parts, because they'll judge voice AI on calls answered, leads qualified and meetings booked, not on how it was put together.
If you'd rather not work out how to build an AI voice agent yourself, hand us the engineering: we build the agent for your business and test it on difficult practice calls before it goes live, the way our own AI outbound sales agent is built, and our AI practice calls train the people who take whatever an agent hands over, yours or ours.
“We're open about who we want to work with: call centres and sales teams that want their people trained, their leads worked and the low-value calls taken off their operators, and businesses that could never afford a call centre but still need their phones answered and their leads called.”
Connect a phone line to speech recognition, an AI model and a synthetic voice, or one speech-to-speech model. Give it one clear job, your information, the tools it needs and rules for handing over. Then test it on hundreds of difficult practice calls before a small, closely watched live pilot.
A speech-to-speech model listens to audio and replies in audio directly, instead of turning speech into text, writing a text reply and turning that into speech. It is usually faster and more natural, but harder to check word by word, and offers fewer voices.
Giving it too many call types at once, testing only friendly calls, measuring delay to a filler word instead of the real answer, letting it say an action is done before your system confirms it, no clean handover to a person, and changing it without rerunning every practice call.
Yes, no-code platforms let you build a voice agent with prompts and settings, and they are a quick way to learn how to build an AI voice agent. Reliable agents for real customers still need a carefully written call, connections to your systems, a handover to people and a great deal of testing, which no tool does for you.
A working demo takes days. A reliable agent that handles real customers, interruptions, accents and edge cases usually takes weeks to months of testing and tuning, followed by a small pilot before it takes all calls.
Running costs are usually charged per minute and cover the phone line, speech recognition, the AI model and the voice, plus any platform fee. Set-up, testing and weekly call reviews cost extra, in staff time or supplier fees.
Saying so plainly, near the start of the call, is good practice, and our agents say they are an AI and that the call is recorded. In the US, the FCC treats AI voices as "artificial" under the TCPA, so consent rules apply; in South Africa POPIA section 69 governs marketing calls. This is general information: take legal advice.
Yes, if it is tested on South African callers. Research on African-accented English found off-the-shelf speech recognition makes several times more errors than on standard benchmarks. Test with your own callers' accents, language-switching and noisy cell lines, and choose a voice that sounds local.
You keep the script, the rules and the information; we build, test and run the agent with you.