speech-to-speech

Product Introduction

1. Definition: Speech-to-speech is an open-source, modular software pipeline for building low-latency voice agents. Technically, it is a cascading system that integrates Voice Activity Detection (VAD), Automatic Speech Recognition (STT), a Large Language Model (LLM), and Text-to-Speech (TTS) synthesis into a single, cohesive application.
2. Core Value Proposition: It solves the critical problems of privacy, vendor lock-in, and high costs associated with cloud-based voice APIs by enabling developers to create fully offline, self-hosted, and customizable voice assistants using state-of-the-art open-source AI models.

Main Features

1. Modular, Swappable Pipeline: The core architecture is a four-stage cascade (VAD -> STT -> LLM -> TTS) where each component runs in its own thread and is connected via queues. Every stage is swappable, allowing developers to mix and match different open-source models (e.g., Parakeet TDT for STT, Qwen3-TTS for synthesis) or commercial APIs based on their specific needs for language support, latency, or quality.
2. OpenAI Realtime-Compatible API: The system exposes its functionality through a WebSocket server that implements the OpenAI Realtime API protocol. This allows any client application built for OpenAI's voice API to connect seamlessly to this local, private backend, ensuring broad compatibility and easy integration into existing projects.
3. Multi-Backend Support for Local Inference: It provides extensive support for running models locally on various hardware. Key backends include Silero VAD v5 for voice detection; Parakeet TDT, Whisper (via Transformers, Faster Whisper, or MLX), and Paraformer for STT; the transformers library and mlx-lm for local LLM inference on CUDA/CPU and Apple Silicon respectively; and Qwen3-TTS (GGML/CUDA/MLX), Kokoro, Pocket TTS, and ChatTTS for speech synthesis.
4. Flexible LLM Integration: The LLM slot is highly flexible, speaking OpenAI-compatible protocols. It can be pointed at hosted providers (OpenAI, Hugging Face Inference Endpoints, OpenRouter), self-hosted servers (vLLM, llama.cpp), or run entirely in-process using transformers or mlx-lm, enabling a spectrum from fully cloud-based to fully local operation.
5. Multi-Language and Multi-Mode Operation: The pipeline supports multilingual interactions by leveraging the capabilities of its chosen STT and TTS backends. It operates in several modes: realtime (WebSocket API), local (direct microphone/speaker use), websocket (raw PCM), and socket (TCP), catering to different deployment scenarios from direct desktop use to client-server applications.

Problems Solved

1. Pain Point: Privacy and Data Sovereignty Risk. Cloud-based voice assistants require sending sensitive audio data to third-party servers. Speech-to-speech eliminates this risk by enabling fully offline, on-premises deployment where all audio processing and conversation data remain on the user's own hardware.
2. Pain Point: Vendor Lock-in and High API Costs. Reliance on proprietary cloud APIs creates dependency and recurring expenses. This platform provides an open-source alternative, allowing organizations to avoid subscription fees and maintain control over their voice agent stack, using either free open-source models or their own fine-tuned versions.
3. Target Audience: The primary users are Developers and ML Engineers building custom voice applications, Researchers experimenting with speech AI pipelines, Robotics Companies (like Pollen Robotics for their Reachy Mini), and Businesses in healthcare, finance, or government that require private, customizable voice interaction without external data leaks.
4. Use Cases: Essential scenarios include building private voice assistants for smart homes or offices, creating interactive voice agents for customer service or education, integrating conversational AI into robots and IoT devices, and developing assistive technology where user data confidentiality is paramount.

Unique Advantages

1. Differentiation: Unlike monolithic cloud services (Google Dialogflow, Amazon Lex) or single-model toolkits, Speech-to-speech is a fully modular, open-source pipeline. It differs from other local solutions by offering a production-ready, low-latency architecture with direct compatibility with the emerging OpenAI Realtime standard, making it both powerful and easy to adopt.
2. Key Innovation: Its primary innovation is the seamless integration of best-in-class open-source speech AI models into a standardized, swappable, and low-latency pipeline. The focus on the OpenAI Realtime API protocol as the interface is a strategic choice that provides immediate utility and future-proofing, bridging the gap between cutting-edge open-source research and practical, deployable applications.

Frequently Asked Questions (FAQ)

1. Can I run Speech-to-speech completely offline without any internet connection? Yes, Speech-to-speech is designed for fully offline operation. You can use local STT models like Parakeet TDT, run an LLM with llama.cpp or the transformers library, and use a local TTS like Qwen3-TTS, creating a voice agent that requires no external API calls.
2. What hardware is needed to run Speech-to-speech locally? Requirements vary by model choice. For a lightweight setup, a modern CPU can run smaller models. For optimal performance with larger models (e.g., 7B+ parameter LLMs), a machine with a capable NVIDIA GPU (for CUDA) or Apple Silicon (for MLX acceleration) is recommended. The pipeline is modular, allowing you to choose models that fit your hardware constraints.
3. How does Speech-to-speech handle different languages? Multilingual support depends on the selected backends. You can use Whisper for multilingual STT, pair it with a multilingual LLM, and use a TTS like Qwen3-TTS or MMS TTS that supports multiple languages. The --language auto flag enables automatic language detection per utterance.
4. Is Speech-to-speech suitable for commercial and production use? Yes, it is licensed under the Apache 2.0 license, permitting commercial use. It is battle-tested, running in production as the conversation backend for thousands of Reachy Mini robots, demonstrating its robustness for real-world applications.
5. How do I connect my existing application to the Speech-to-speech server? If your application uses the OpenAI Realtime API SDK, you can simply change the base URL to point to your Speech-to-speech server (e.g., ws://localhost:8765/v1). For custom clients, you can use the simpler websocket or socket modes for raw audio streaming.