speech-to-speech v1.0.0
The first major release brings a single Realtime engine, explicit server and
microphone-client commands, more speech backends, and improved conversation
lifecycle handling. These notes cover changes since v0.2.12.
Highlights
- Three commands:
serve, talk, and local. Run the Realtime server,
connect the packaged microphone/speaker client, or start both in one terminal.
The client also supports local tool execution with --tool-module.
- More speech backends. Add Qwen3-ASR speech recognition, OpenAI-compatible
STT and TTS endpoints, stateful OpenAI Realtime transcription, experimental
vLLM Realtime transcription, and optional OmniVoice and Supertonic TTS.
- More reliable realtime conversations. Improve ordered text, audio, and
tool-call output; transcript and content-part events; interrupted responses;
remote-request cancellation; and session cleanup. The browser demo uses the
OpenAI Agents SDK, with integration tests for its WebSocket and WebRTC
transports against the implemented core Realtime event set.
- Updated Apple Silicon setup. Refresh the MLX dependencies and default to
the 4-bit mlx-community/Qwen3-4B-Instruct-2507-4bit LLM with 6-bit Qwen3-TTS.
The Mac preset supplies defaults while respecting explicit overrides.
- Clearer setup and diagnostics. The README provides local Mac, local
NVIDIA, and hosted-LLM starting configurations, a shared llama.cpp recipe,
offline setup guidance, and a speaker-feedback workaround. Full transcript
logging is now opt-in with --log_transcripts; operational logs omit
conversation text by default.
Upgrading from v0.2.12
In your Python environment:
pip install --upgrade "speech-to-speech==1.0.0"
Update scripts and service definitions to use the explicit commands:
| Previous invocation |
v1.0.0 invocation |
speech-to-speech |
speech-to-speech serve |
speech-to-speech --mode realtime |
speech-to-speech serve |
speech-to-speech --mode local |
speech-to-speech local |
speech-to-speech --local_mac_optimal_settings |
speech-to-speech local --mac-optimal-settings |
python scripts/listen_and_play_realtime.py --host 127.0.0.1 --port 8765 |
speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime |
The --mode realtime and --mode local forms still work temporarily with a
deprecation warning. Other mode values, including socket and raw-websocket,
have been removed. Migrate raw PCM clients to the Realtime WebSocket or WebRTC
API; the old raw transport flags are no longer accepted.
The --mac-optimal-settings preset no longer selects a command: use local
for microphone/speaker interaction or serve for a server. serve binds to
127.0.0.1 by default; set --host explicitly for network access. local
uses loopback only.
The default hosted LLM is now gpt-5.6-terra, with reasoning effort none.
Pin --model_name if your deployment depends on a particular model. Local
Mac users can likewise override the preset's model with --model_name.
The redundant facebook-mms, language-detection, and websocket installation
extras were removed; their dependencies are included in the standard package.
Install speech-to-speech[webrtc] for WebRTC, speech-to-speech[omnivoice] for
OmniVoice, or speech-to-speech[supertonic] for Supertonic. Known options for
inactive backends are accepted but ignored with a warning.
Compatibility and setup
- Python 3.10+ is supported; the installation smoke tests use Python 3.11 on
Linux and Apple Silicon. Check the README's CUDA wheel guidance for Linux.
- Realtime compatibility covers the documented core event set, not every
OpenAI Realtime API feature. See the
protocol reference.
--local_audio_block_mic_during_playback pauses microphone capture during
assistant playback to reduce speaker feedback. It disables spoken
interruptions during playback; it does not perform acoustic echo cancellation.
See the README for starting configurations and the
full changelog
for all changes and contributors.
详细ChangeLog
speech-to-speech 0.2.12 is the final planned release in the 0.2.x line before
the next round of larger changes. It brings smarter turn-taking, WebRTC support,
direct audio input for audio-capable LLMs, more complete OpenAI Realtime protocol
behavior, and a significantly improved browser demo.
Highlights
- Smarter endpointing with Smart Turn v3.2. Realtime mode now enables the
quantized CPU Smart Turn model by default to distinguish completed turns from
mid-thought pauses while speculative STT and LLM work continues. Use
--no_smart_turn to retain Silero-only endpointing. (#192)
- WebRTC transport for the OpenAI Realtime API. Install the new
webrtc
extra to use SDP negotiation at POST /v1/realtime/calls, RTP audio, and the
oai-events data channel alongside the existing WebSocket transport. Both
transports share the same pipeline pool, event dispatch, cancellation, and
interruption behavior. (#352)
- Direct audio input for audio-capable LLMs. Run with
--stt none, the
Chat Completions backend, and an explicitly selected audio-capable model to
send completed VAD audio directly to the model. Audio history, cancellation,
tool calls, usage accounting, and provider failures are handled transactionally.
(#298)
- Optional remote-LLM proxy.
--enable_llm_proxy exposes the configured
remote backend through the matching /v1/responses or
/v1/chat/completions path, with streaming passthrough and usage accounting.
The engine deliberately does not authenticate or throttle these routes, so
standalone deployments must keep them on a trusted network or behind an
authenticated gateway. (#368)
Realtime, demo, and backend improvements
- The browser demo can select microphone and speaker devices, start with a
configurable assistant greeting, forward signed-in Hugging Face identity for
session allocation, show user-turn lifecycle feedback, and replay the exact
captured user audio locally. (#376,
#389,
#390)
- Qwen3-TTS GGML users can select quantization, load local talker and codec GGUF
files, configure voice-reference caching, and reuse precomputed .spk and
.rvq references. (#403)
- Successful
session.update requests now receive the expected
session.updated event with the effective session configuration.
(#413,
#417,
#418)
- Language prompting now covers every language reported by the bundled STT
handlers, including the full default Parakeet TDT language set.
(#395)
- A new guide documents a local Gemma 4 12B Realtime setup on Apple Silicon,
including native-audio bypass, memory tuning, cancellation, barge-in, and
troubleshooting. (#422)
Reliability and security
- Session teardown now preserves drain sentinels, releases capacity after setup
failures, quarantines stuck pipeline units, reports them through /v1/pool,
and prevents stale teardown signals from releasing the wrong session.
(#358)
- Speculative turn tracking no longer resurrects untracked revisions after a
reset or LRU eviction, preventing reused turn IDs from suppressing responses.
(#391)
- Both MLX Audio Whisper and Lightning Whisper MLX now use the global MLX lock,
preventing Metal command-buffer crashes when STT overlaps other MLX work on
Apple Silicon. (#379,
#387)
- Optional handler imports no longer reconfigure the host application's root
logger, and setup/cleanup output now uses module loggers.
(#409)
- NLTK is upgraded to 3.10.0 to address CVE-2026-54293 / GHSA-p4gq-832x-fm9v.
(#400)
Breaking change
The raw PCM WebSocket mode is now named raw-websocket. Commands using
--mode websocket must change to --mode raw-websocket; the old value is no
longer accepted. Realtime mode is unchanged and still supports both WebSocket
and WebRTC transports. (#401)
Packaging notes
- Smart Turn adds
huggingface-hub and onnxruntime as standard dependencies.
- WebRTC support is available through
pip install "speech-to-speech[webrtc]".
- The Smart Turn model is downloaded from the Hugging Face Hub on first use
unless a local model path is supplied.
All merged changes
New contributors
Thank you to @dcolley, @Hotragn, @varunsahni18, @ztcools,
@salignatmoandal, and @hrqiang for their first contributions to the project.
Full changelog: v0.2.11...v0.2.12
详细ChangeLog
