Building voice agents

How to build an AI voice agent: the parts, the architecture, the steps and the testing

The short answer

How to build an AI voice agent, in short: connect a phone line to speech recognition, an AI model and a synthetic voice (or one speech-to-speech model), give it one clear job, your information and tools, then test it on hundreds of difficult practice calls before a small live pilot. A demo takes days; a reliable agent takes months.

By , founder and CEO of ShiftMateUpdated 14 min read
A team planning how to build an AI voice agent, from the parts to testing and launch
Mike Steenkamp, founder and CEO of ShiftMate, a South African voice agent specialist
“ShiftMate is a South African voice agent specialist, and we've built the parts this guide describes ourselves, including hundreds of test calls to get the delay right. Here's how to build an AI voice agent: the parts, the two main ways to put them together, the steps, and when building isn't worth it.”
Mike Steenkamp, founder and CEO of ShiftMate, Umhlanga, South Africa
Key takeaways
  • Knowing how to build an AI voice agent is mostly knowing how to test one: the demo is the easy tenth. Start with our complete guide to AI voice agents if you're new to them.
  • Every agent has the same eight parts: phone line, speech recognition, turn-taking, AI model, knowledge and tools, voice, guardrails and monitoring.
  • Pipeline (speech to text, AI, voice) gives you control; speech-to-speech gives you speed. Many working systems mix the two.
  • Conversation gaps between people are often around a fifth of a second (PNAS, 2009), so measure delay to the real answer, from where your callers are.
  • Build it yourself only if voice is your product. Otherwise use a platform or have it built, and keep ownership of the testing.

How to build an AI voice agent: what you are really building

An AI voice agent is software that holds a phone conversation: it listens, works out what the caller wants, does something useful (books, looks up, records, hands over) and replies in a natural voice, fast enough that the conversation feels normal. Learning how to build an AI voice agent means learning to make those pieces work together in real time, on real phone lines, with real callers who don't follow your script.

The good news: the parts are widely available, and a developer can make a working demo in days. The harder news: a demo isn't an agent you can trust with customers. Most of the work is in the calls that go wrong, which is why this guide spends as much time on testing and launch as on the build itself. If you want the honest view of the effort first, read how hard it is to build a voice agent.

It explains why so many projects stall. In a Gartner survey of 187 customer service leaders in mid-2024, 44% were exploring a voicebot and 11% piloting one, but only 5% had one deployed. The gap between those numbers is mostly testing, connections to real systems and trust, not the AI.

What are the parts of an AI voice agent?

The first lesson in how to build an AI voice agent is that every agent has the same eight parts, whether you assemble them yourself or a platform hides them behind one screen. Knowing them tells you where problems will come from.

PartWhat it doesWhat goes wrong
Phone line or web callCarries the audio: a phone number, a connection to your existing phone system, or a browser callPoor audio, delay on long routes, recording consent
Speech recognitionTurns the caller's words into textAccents, noise, names, numbers and addresses
Turn-takingDecides when the caller has finished, and stops the agent when they cut inTalking over people, or awkward silences
AI language modelUnderstands the request and decides what to say and doMaking things up, going off script, slow replies
Knowledge and toolsYour information, plus actions such as booking, looking up an order or sending an SMSOutdated facts; actions it shouldn't take
Synthetic voiceTurns the reply into speechRobotic delivery, wrong pronunciation of local names
GuardrailsChecks what the agent will say and do against your rulesUnapproved promises, prices and advice
MonitoringRecordings, transcripts, outcomes, scores and alertsProblems found by customers instead of by you

Two parts are often forgotten in early builds and cause the most trouble later: turn-taking and guardrails. Turn-taking decides whether the call feels like a conversation or a walkie-talkie. Guardrails decide whether the agent can be trusted to speak for your business. Plan both from day one.

Pipeline or speech-to-speech: which voice AI architecture should you choose?

There are two main ways to put the parts together, and the choice of voice AI architecture is the biggest early decision in how to build an AI voice agent: it shapes speed, control and cost. A pipeline runs three steps one after another: speech recognition turns the caller's words into text, an AI model writes a text reply, and a synthetic voice speaks it. A speech-to-speech model listens to the audio and replies in audio directly, in one step.

Pipeline (speech to text, AI model, voice)Speech-to-speech (one model)
SpeedEach step adds delay, so it needs careful tuningUsually faster, with more natural turn-taking
ControlYou can read and check every reply before it is spokenHarder to check each word before the caller hears it
Voice choiceAny voice, including local South African voicesLimited to the voices the model offers
Swapping partsChange one part without rebuilding the restAll or nothing
Tone and emotionLost in the text step unless you add it backCan carry tone from the caller's voice
Best forStrict scripts, regulated calls, local voicesNatural, fast, open conversations

Many working systems mix the two: a fast model handles the conversation while a separate check reviews anything risky, or a pipeline handles regulated calls while a speech-to-speech model handles open chat. There is no single right answer. Choose by asking three questions: how strict must the script be, how fast must the reply be, and which voice will your callers trust?

Where the delay comes from

People expect a reply quickly: in normal conversation the gap between one person finishing and the other starting is often around a fifth of a second (Stivers and others, PNAS 2009). An agent spends time deciding the caller has finished, understanding them, writing a reply, starting the voice, and sending audio across the phone network. Each slice looks small; together they decide whether the call feels natural. Measure the whole round trip, on real phone lines, from where your callers are.

What we learned: Speed (latency)
“We learned to measure the real answer, not the filler. A quick "Sure" makes a speed number look good while the caller is still waiting for what they asked. And we measure the round trip from a South African caller, not from a server next door to the AI.”
Mike Steenkamp, founder and CEO of ShiftMate

How do you build an AI voice agent, step by step?

Here is how to build an AI voice agent from idea to live calls. The order matters: the steps people skip (writing the call down, testing, the pilot) are the ones that decide whether customers trust it.

  1. 1

    Pick one job with a clear outcome

    Book an appointment, follow up a lead, confirm a delivery, give a claim status. Not "answer all our calls". Write down what a successful call ends with.

  2. 2

    Study your real calls

    Listen to recordings of the call type you're automating: how callers open, the questions they ask, the accents, the complaints, the ways calls go wrong. These become your test callers later.

  3. 3

    Write the call down

    The stages, the facts the agent may use, the questions it must ask, what it must never say, and exactly when it hands over to a person.

  4. 4

    Choose the architecture and the voice

    Pipeline, speech-to-speech or a mix, and a voice your callers will trust. Write an opening that says it's an AI and that the call is recorded.

  5. 5

    Connect only the tools the job needs

    Booking system, customer records, order lookup. Give it as little access as the job allows, and make it confirm actions before it tells the caller they're done.

  6. 6

    Add guardrails and the handover

    Rules it can't break whatever the caller says, and a handover that passes a summary to the person, so the caller never starts again.

  7. 7

    Test on difficult practice calls

    Hundreds of them: interruptions, accents, noise, two problems in one call, angry callers, people who ask for a person. Each call passed several times, not once.

  8. 8

    Pilot with a small share of real calls

    A few percent of calls, with people reviewing recordings every day and a quick way to switch the agent off.

  9. 9

    Measure, fix and retest

    Track resolution, handovers, repeat calls and complaints, not containment (containment vs resolution explains why). Turn each failure into a new practice call and rerun the full set before every change goes live.

What we learned: Honesty
“Our agents say they're an AI and that the call is recorded. People forgive a machine that's honest with them; they don't forgive one that pretends.”
Mike Steenkamp, founder and CEO of ShiftMate
What we learned: Trust
“A spoken promise isn't a completed action. Our agents only tell a caller something is done once the system confirms it is done, not because the conversation went well.”
Mike Steenkamp, founder and CEO of ShiftMate

How do you test a voice agent before it speaks to customers?

You test a voice agent with a library of difficult practice calls, scored against pass marks you set before you see results, and rerun on every new version. Any honest guide to how to build an AI voice agent puts the effort here: the happy path takes an afternoon; the other calls take months.

Practice callerWhat it testsPass when the agent...
Interrupts mid-sentenceTurn-takingStops, listens, carries on without repeating itself
Two problems in one breathKeeping trackDeals with both, or hands over with both noted
Strong local accent on a noisy lineUnderstandingReads names and numbers back correctly
Asks something outside its informationMaking things upSays it doesn't know, offers a person
"Am I talking to a robot?"HonestySays plainly that it's an AI
"Put me through to a person"HandoverHands over at once, with a summary
"Take me off your list"Opt-outsStops selling, confirms, ends politely
Angry or distressedCareSlows down, acknowledges, hands over to a trained person
What we learned: Testing before launch
“Before a version of one of our AI sales agents may phone anyone, it has to pass every practice situation three times: the busy prospect, the one who only wants an email, the one who asks if it's a robot, the one who says take me off your list. Passing once proves nothing.”
Mike Steenkamp, founder and CEO of ShiftMate

Test every call type on its own: a rule that is right for an outbound call can be wrong for an inbound one. And keep every failed call as a permanent test, so a fix in one place can't quietly break another. Our guide to why AI voice agents fail has a full 20-call test plan and the numbers to track.

How do you launch and run a voice agent in South Africa?

The last stage of how to build an AI voice agent is the launch: start small, watch closely, and grow only when the numbers hold. Start with a few percent of calls or one lead source, with a named owner who reviews flagged calls daily. Expand when resolution and complaints are steady for several weeks, not when the first week looks good.

South Africa adds three things to plan for. First, accents and languages: research on African-accented English (AfriSpeech-200, 2023) found off-the-shelf speech recognition makes several times more errors on these accents than on the standard benchmarks, so test with your own callers and with language-switching. Second, voice: callers trust a voice that sounds like them. Third, the law.

What we learned: South African voices
“The market responds better to natural, authentic South African voices. An American or British voice on a South African line sounds like a scam call. So our sales agents always sound South African, from the first word to the last, even on UK or US campaigns.”
Mike Steenkamp, founder and CEO of ShiftMate

For outbound calls, POPIA section 69 requires consent for direct marketing by electronic communication, including automatic calling machines, with a narrow exception for your own existing customers. South Africa's national opt-out registry is being phased in: direct marketers register with the National Consumer Commission from 15 September 2026, cleanse their lists monthly (free cleansing December 2026 to April 2027), and consumers can block marketers from May 2027 (NCC). Calls to consumers at home, unless they asked to be contacted, keep to set hours: no Sundays or public holidays, Saturdays 09:00 to 13:00, other days 08:00 to 20:00. In the US, the FCC ruled in February 2024 that AI voices count as "artificial" under the TCPA, so consent rules apply. This is general information: take legal advice for your own calls.

Budget for running costs too. Most platforms charge per minute, and the all-in figure includes the phone line, speech recognition, the AI model and the voice, plus set-up and the time spent testing. Our AI voice agent pricing guide breaks down what each part costs.

Should you build it yourself, use a platform or have it done for you?

Before you decide how to build an AI voice agent, decide who should build it. There are three ways to get an agent, and the right one depends on whether voice is your product or a channel for your business.

Build it yourselfDeveloper or no-code platformDone for you
You needEngineers who know real-time audio, phone systems and AISomeone technical to configure, connect and testYour scripts, rules, information and time to review calls
Time to a demoWeeksDaysDays to weeks
Time to reliableMonthsWeeks to monthsWeeks
ControlFullWithin the platform's limitsAgreed with the supplier
Who tests and tunes itYouYouThe supplier, with you
Best forVoice is your productTech teams with time to iterateBusinesses that want the outcome, not a project

Whichever route you choose, keep ownership of three things: the definition of a good call, the practice-call library, and the right to see every recording. A supplier who won't let you run your own test calls before you sign is telling you something.

When not to build an AI voice agent at all

Sometimes the best answer to how to build an AI voice agent is: don't, yet. Skip it, or wait, when:

  • Your people can't resolve the call type either, because the process or systems are broken. Fix that first.
  • The calls need judgement, comfort or advice only a licensed person may give: bereavement, distress, financial advice.
  • You get too few calls for the set-up and testing effort to pay back. A good website chat or callback form may do.
  • Nobody can own the agent's quality after launch, with time each week to review calls.
  • You can't get consent for outbound calls, or can't keep to the opt-out registry.
  • Your callers mostly use a language or accent no supplier has proven on your own recordings.

A voice agent doesn't replace training your people either. When it takes the simple calls, the ones left for people are harder, and they need practice for them.

Build or buy: our view

Our view on build or buy is simple. If voice is your product, build it. If voice is a channel for your business, we believe your team's time is better spent on the script, the rules and the people than on audio engineering. We think more businesses will buy the result rather than build the parts, because they'll judge voice AI on calls answered, leads qualified and meetings booked, not on how it was put together.

If you'd rather not work out how to build an AI voice agent yourself, hand us the engineering: we build the agent for your business and test it on difficult practice calls before it goes live, the way our own AI outbound sales agent is built, and our AI practice calls train the people who take whatever an agent hands over, yours or ours.

What we learned: Who we want to work with
“We're open about who we want to work with: call centres and sales teams that want their people trained, their leads worked and the low-value calls taken off their operators, and businesses that could never afford a call centre but still need their phones answered and their leads called.”
Mike Steenkamp, founder and CEO of ShiftMate

Questions people ask

How do you build an AI voice agent?

Connect a phone line to speech recognition, an AI model and a synthetic voice, or one speech-to-speech model. Give it one clear job, your information, the tools it needs and rules for handing over. Then test it on hundreds of difficult practice calls before a small, closely watched live pilot.

What is speech-to-speech in voice AI?

A speech-to-speech model listens to audio and replies in audio directly, instead of turning speech into text, writing a text reply and turning that into speech. It is usually faster and more natural, but harder to check word by word, and offers fewer voices.

What are the most common mistakes when building a voice agent?

Giving it too many call types at once, testing only friendly calls, measuring delay to a filler word instead of the real answer, letting it say an action is done before your system confirms it, no clean handover to a person, and changing it without rerunning every practice call.

Can I build a voice agent without coding?

Yes, no-code platforms let you build a voice agent with prompts and settings, and they are a quick way to learn how to build an AI voice agent. Reliable agents for real customers still need a carefully written call, connections to your systems, a handover to people and a great deal of testing, which no tool does for you.

How long does it take to build an AI voice agent?

A working demo takes days. A reliable agent that handles real customers, interruptions, accents and edge cases usually takes weeks to months of testing and tuning, followed by a small pilot before it takes all calls.

What does it cost to build and run an AI voice agent?

Running costs are usually charged per minute and cover the phone line, speech recognition, the AI model and the voice, plus any platform fee. Set-up, testing and weekly call reviews cost extra, in staff time or supplier fees.

Does an AI voice agent have to say it is an AI?

Saying so plainly, near the start of the call, is good practice, and our agents say they are an AI and that the call is recorded. In the US, the FCC treats AI voices as "artificial" under the TCPA, so consent rules apply; in South Africa POPIA section 69 governs marketing calls. This is general information: take legal advice.

Can an AI voice agent work for South African callers?

Yes, if it is tested on South African callers. Research on African-accented English found off-the-shelf speech recognition makes several times more errors than on standard benchmarks. Test with your own callers' accents, language-switching and noisy cell lines, and choose a voice that sounds local.

Want the outcome without the project?

You keep the script, the rules and the information; we build, test and run the agent with you.