Voice AI best practices

Start with the outcome your assistant must achieve: answer a question, book an offered appointment, qualify a lead, solve a support issue, or connect the caller to a person. A fast answer is useful only when it is correct and the requested action actually completes.

Choose the engine for the job

Engine	Good starting point	What to verify
Pipeline	Workflows that need explicit speech recognition, response generation and voice control	Recognition of names and numbers, response timing, tool results and voice-language compatibility
Realtime	Fluid spoken conversations with native audio understanding and speech	Spoken response latency, interruption recovery, tool execution and grounded answers
Half-Cascade	Native audio understanding with a separately chosen speaking voice	Native-model response timing plus the selected voice’s synthesis and playback
Full Duplex	Conversations that need overlapping speech and backchannels	Whether useful interruptions are handled, information survives overlaps, and required announcements finish
Translate	Conversations between people who speak different languages	Meaning and entity preservation in both directions, speaker attribution and translation delay

Engine and model availability depend on your plan and workspace. Full Duplex requires workspace beta access. See Engine modes and Translate.

Prioritize the right outcomes

Use case	Prioritize	A successful test proves
Customer service	Correct policy answers, recognition of identifiers, and appropriate escalation	The issue is resolved or an appropriate handoff reaches its destination
Appointment booking	Offered slots, timezone clarity, explicit confirmation and valid tool arguments	The booking exists for the confirmed time; an unoffered time is rejected
Sales	Accurate product facts, relevant qualification and consent	The agreed next step is recorded without invented prices or commitments
Technical support	Accurate error codes, concise instructions and safe escalation	The stated issue is addressed; missing knowledge is acknowledged

Do not choose one universal winner from unrelated tasks. Compare configurations on the same scenarios, languages and success criteria, then select the one that performs well on your own call mix.

Measure the conversation the caller hears

Metric	Definition	Reading the result
End-to-end latency	Caller speech end to the first meaningful response audio received	Includes the conversation path; report overlaps separately
P50 / P95 latency	Median and 95th percentile of eligible measured responses	P50 describes a typical turn; P95 describes slower turns in the sample
LLM TTFT	Generation request to first response text token	Useful for Pipeline; native audio token timing is a different metric
TTS TTFB	Synthesis request to first generated audio	A component measure; network playback adds further delay
End-of-turn delay	Caller speech end to committing the turn	Includes recognition and turn detection; do not add recognition delay twice
Task success	Successful outcomes divided by eligible attempted tasks	Verify the final booking, record, answer or handoff
Containment	Eligible tasks resolved without a human divided by eligible tasks	A correct requested escalation remains a good outcome but is not contained
Tool success	Correct completed executions divided by attempted executions	Check arguments, returned results and resulting state
Interruption handling	Yield timing and correct recovery after eligible interruptions	Track false interruptions and lost information separately
STT accuracy / WER	Word edits divided by reference words in independently transcribed audio	Also check critical names, amounts and times; a low overall WER can hide entity errors
Call drop rate	Unexpected disconnections divided by established test calls	Exclude planned hangups and classify failed connection attempts separately
Hallucination rate	Answers containing a material unsupported claim divided by evaluated answers	Use explicit reference facts and review disputed cases
Transfer success	Confirmed destination connections divided by attempted transfers	A request, announcement or accepted transfer instruction alone does not prove connection

Native Realtime engines do not provide the same separate text-generation and speech-synthesis metrics as Pipeline. Show unavailable values as not applicable. They are not zero-latency results.

Build a useful test set

Use the same reference audio and expected outcomes for each comparable configuration. Include both ordinary and difficult calls: missing required details, empty lookup results, unavailable appointments, caller corrections, short acknowledgments, interruptions, silence and requests outside the assistant’s scope. Test each language independently. Check spelling, dates, timezones, amounts and identifiers with a known reference. For speech recognition, use recordings with independently verified transcripts. Synthetic voices are useful for controlled conversation tests, but they do not represent every accent, microphone or real caller. Repeat scenarios and rotate configuration order. Keep language, prompt, tools, region, transport and audio conditions consistent. Compare clear audio separately from noise, weak microphones and packet loss. Report the number of calls and the number of valid observations for each metric. A short screening run can identify compatibility failures and candidates for further testing. It does not establish a reliable population P95 or a rare failure rate. Publish sample counts, uncertainty and exclusions with any ranking. Separate observations from existing live calls from controlled comparisons, because their tasks and configurations can differ.

Improve one part at a time

Keep the prompt focused on the assistant’s role, known facts and action preconditions.
Use short spoken answers and ask for one missing detail at a time.
Offer only tools relevant to the current task and confirm their final results.
Match the voice to the language and listen to names, numbers and required announcements.
Adjust response timing with pauses and interruptions in the test set. Faster turn detection can cut off callers.
Re-run the same scenarios after changing a model, prompt, voice or integration.

Use Simulations, AI quality assurance and the call history to inspect outcomes. For privacy, use consented recordings or controlled test data and share aggregate results rather than customer conversations.

Interpret cost alongside success

Compare the complete configuration’s usage and your applicable tariff. A cheaper minute can cost more per successful outcome when it causes repeated calls, failed actions or unnecessary escalation. Missing price or usage information is unknown, not free. Supplier billing and internal operating-cost calculations belong in your private evaluation; see Usage and billing for customer-facing consumption.

Getting started

Assistants

Flow builder

Telephony

Channels

Audience

Campaigns

Live operations

Workspace settings

Billing

Voice AI best practices

Choose the engine for the job

Prioritize the right outcomes

Measure the conversation the caller hears

Build a useful test set

Improve one part at a time

Interpret cost alongside success

​Choose the engine for the job

​Prioritize the right outcomes

​Measure the conversation the caller hears

​Build a useful test set

​Improve one part at a time

​Interpret cost alongside success

Choose the engine for the job

Prioritize the right outcomes

Measure the conversation the caller hears

Build a useful test set

Improve one part at a time

Interpret cost alongside success