voice agent
In plain wordsAn AI system you talk to out loud, for example on a phone call, that listens, answers in a natural-sounding voice and can carry out tasks such as booking appointments or checking an order.
An AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. that converses with people in real time through speech, typically over telephone or app audio, and can take actions through toolstool use, n.: An AI model's ability to reach beyond its own knowledge by operating outside tools — searching the web, running code, using a calculator or an app — and folding the results into its work. such as calendars, databases or payment systems. Two architectures dominate. A cascaded pipeline chains voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk., streaming speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video., a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. and text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps., which keeps each component replaceable and the conversation inspectable as text; end-to-end speech-to-speech modelsspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice. work on audio directly, preserving tone and cutting delay. The hardest engineering problems are conversational rather than linguistic: keeping response delay near the few hundred milliseconds of human turn-taking, deciding when the caller has finished speaking (endpointing), handling interruptions ('barge-in') and overlapping speech, recovering from recognition errors, and integrating with telephony systems. Deployments include customer service, scheduling, sales, interviewing and accessibility services, and raise issues of disclosure, consent, voice cloningvoice cloning, n.: Using AI to copy a specific person's voice from recordings, so a computer can say new things that sound like them — useful for dubbing and accessibility, and abused in scams. and fraud.
Our voice agent answers after-hours calls, checks the order system and hands off to a human if the caller asks.
Origin
In call-center jargon a 'voice agent' was a human who handles phone calls. The AI sense spread in 2024 with low-latency speech models such as OpenAI's GPT-4oGPT-4o, prop. n.: An OpenAI model from 2024 that handles text, images and speech in one system, enabling fast, natural voice conversations. Many users grew attached to its personality, which made its retirement controversial. voice mode and Realtime API and Kyutai's full-duplex Moshi, and with a wave of start-ups building phone agents. Precursors include interactive voice response systems and Google Duplex (2018), which booked restaurant tables by phone.
Usage note
Broader than a virtual assistantvirtual assistant, n.: Software you speak or type to, like Siri, Alexa or Google Assistant, that answers questions and does simple jobs such as setting timers, playing music or controlling smart-home devices. such as Siri or Alexa, which historically mapped requests to fixed intents. Regulation is spreading: in February 2024 the US Federal Communications Commission ruled that AI-generated voices in robocalls count as 'artificial' under the Telephone Consumer Protection Act, and Article 50 of the EU AI Act requires that people be told when they are interacting with an AI system, unless this is obvious.
Related
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- speech-to-speech modelspeech-to-speech model, n.: An AI model that listens to speech and replies or translates directly in speech, without first writing everything down as text, so conversations feel faster and keep the tone of voice.
- speech recognitionspeech recognition, n.: Technology that turns spoken words into written text, as when a phone transcribes a voice message or captions appear automatically under a video.
- text-to-speechtext-to-speech, n.: Technology that reads written text aloud in a computer-generated voice, used in screen readers, voice assistants, audiobooks and navigation apps.
- voice activity detectionvoice activity detection, n.: Detecting, moment by moment, whether someone is speaking in an audio stream, so a system knows when to listen, when to stop recording and when it is its turn to talk.
- voice cloningvoice cloning, n.: Using AI to copy a specific person's voice from recordings, so a computer can say new things that sound like them — useful for dubbing and accessibility, and abused in scams.
- virtual assistantvirtual assistant, n.: Software you speak or type to, like Siri, Alexa or Google Assistant, that answers questions and does simple jobs such as setting timers, playing music or controlling smart-home devices.
- time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear.
- virtual assistantvirtual assistant, n.: Software you speak or type to, like Siri, Alexa or Google Assistant, that answers questions and does simple jobs such as setting timers, playing music or controlling smart-home devices.
- chatbotchatbot, n.: Software you chat with in ordinary language, by typing or talking, as you would with a person. The newest ones, such as ChatGPT, can answer questions, draft writing and keep long conversations going.
Sources
- Stivers, T. et al. (2009). Universals and Cultural Variation in Turn-Taking in Conversation. PNAS 106(26), 10587–10592.