A real-time, bidirectional channel
The Voice Live API powers voice apps with real-time, two-way communication over a WebSocket connection, exchanging JSON-formatted events. It handles speech recognition and synthesis, supports audio formats like PCM16 and G.711, and adds avatar streaming over WebRTC, built-in noise reduction and echo cancellation, and OpenAI or Azure custom voices.
Key concept · Events and session configuration
Communication is event-based: client events (client→server) like session.update, input_audio_buffer.append, response.create; and server events (server→client) like session.updated, response.done, conversation.item.created. You typically send session.update first to set modalities (text/audio), voice, audio formats and sampling rate, turn detection (Azure semantic VAD with threshold/padding/silence), instructions, and response parameters. Authenticate keyless with Microsoft Entra (Cognitive Services User role, bearer token) — recommended — or with an API key.
Async streaming and interruptions
The Azure AI Voice Live client library for Python is a real-time speech-to-speech client that opens a WebSocket session, streams microphone audio up, and receives server events. As of version 1.0.0 it is async-only — the synchronous API is deprecated, so everything uses async/await.
Key concept · Handling the barge-in
Good event handling is what makes the conversation feel natural. The critical case is a user interruption: on INPUT_AUDIO_BUFFER_SPEECH_STARTED you must immediately stop playback and cancel the current response so the agent doesn't talk over the user. Other key server events: SESSION_UPDATED (session ready — start capturing audio), RESPONSE_AUDIO_DELTA (play this audio chunk), RESPONSE_DONE, and ERROR. A RequestSession carries modalities, instructions, PCM16 audio formats, and a ServerVad turn-detection config.
Configuration that travels with the agent
You can build a Voice Live agent in the Agent playground (voice mode: language, VAD, audio enhancement, voice + tone/rate, interim responses, optional avatar) or in code. Its advantage over raw session code: instructions and configuration are encapsulated in the agent, so the client connects with just an agent ID and complex logic changes without touching client code.
Key concept · The client-app architecture
A recommended split: a VoiceAssistant class (configures/starts the session, processes voice events, manages the agent connection) and an AudioProcessor class (microphone in, speaker out, buffering and playback queues via PyAudio), plus Entra ID authentication. Typical audio settings: 16-bit PCM, mono, 24 kHz, ~50 ms (1,200-byte) chunks; turn detection via Azure semantic VAD multilingual. Note the 512-character metadata limit — chunk configuration across multiple metadata entries.
Chapter 21 quiz
Fourteen questions on the API, the SDK, and the agent. Answer first, then expand Show answer.
Part A · Multiple choice
What transport does the Voice Live API use?
- A A WebSocket connection exchanging JSON events
- B A single REST POST
- C FTP
- D Email
Show answer
A. Real-time, bidirectional, event-based.
Which event should trigger stopping playback and canceling the response?
- A INPUT_AUDIO_BUFFER_SPEECH_STARTED
- B SESSION_UPDATED
- C RESPONSE_DONE
- D ERROR
Show answer
A. The user started speaking — don't talk over them.
As of version 1.0.0, the Python Voice Live SDK is:
- A Async-only (sync deprecated)
- B Sync-only
- C Both, equally supported
- D Deprecated entirely
Show answer
A. Everything uses async/await.
Which event signals the session is ready to start capturing audio?
- A SESSION_UPDATED
- B RESPONSE_AUDIO_DELTA
- C ERROR
- D conversation.item.created
Show answer
A — SESSION_UPDATED.
What is the recommended keyless authentication method?
- A Microsoft Entra (Cognitive Services User role, bearer token)
- B An anonymous connection
- C A shared account password
- D No auth
Show answer
A. API key is the alternative.
What is the main advantage of using an agent over raw session code?
- A Configuration is encapsulated; the client connects with just an agent ID
- B It removes the need for audio
- C It's the only way to authenticate
- D It disables interruptions
Show answer
A. Logic changes without touching client code.
Part B · True or false
Voice Live supports avatar streaming over WebRTC.
Show answer
True. With configurable resolution, bitrate, codec, and visemes/blendshapes.
Turn detection uses Azure semantic VAD with configurable threshold and silence duration.
Show answer
True. Plus prefix padding.
The session.update event is typically sent last, after audio streaming ends.
Show answer
False. It's typically sent first, to configure the session.
Agent metadata has a 512-character limit, so config may need to be chunked.
Show answer
True. Split configuration across multiple metadata entries.
Built-in noise reduction and echo cancellation are available in Voice Live.
Show answer
True. e.g. azure_deep_noise_suppression and server_echo_cancellation.
Part C · Short answer
Name the two classes in the recommended client architecture and their roles.
Show answer
VoiceAssistant (session/agent management) and AudioProcessor (mic in, speaker out, buffering via PyAudio).
Give the typical audio settings (format, channels, sample rate).
Show answer
16-bit PCM, mono, 24,000 Hz (≈50 ms / 1,200-byte chunks).
Name two client events and two server events.
Show answer
Client: session.update, input_audio_buffer.append, response.create. Server: session.updated, response.done, conversation.item.created.