Speech Core

📖 Read in: English · 中文 · 日本語 · 한국어 · Español · Deutsch · Français · हिन्दी · Português · Русский · العربية · Tiếng Việt · Türkçe · ไทย

On-device speech infrastructure in C++17 for Linux, Windows, and Android: voice activity detection, batch and real-time streaming speech-to-text, speaker diarization, text-to-speech, and the voice-agent pipeline that connects them.

Runs locally on CPU. No cloud, no Python at inference, and no audio leaves the machine.

📚 Full documentation → · 🐧 Linux · 🪟 Windows · ⌨️ Desktop CLI · 🔊 HTTP audio

🤗 Models · 🍎 Apple sibling · 💬 Discord

Demo

A full offline voice agent in 1.2 GB on Android — the speech-android control-demo

Why speech-core

speech-core separates a small, model-agnostic orchestration layer from optional inference backends. The core owns turn detection, interruption handling, audio utilities, conversation state, and tool calls; your application chooses the models.

• Local-first: pure C++17 core, float audio buffers, no network or platform audio dependency.
• Built for live agents: VAD-driven turns, eager STT, partial transcripts, barge-in, streaming TTS, and tool calling.
• Real streaming ASR: cache-aware RNN-T decoders, end-of-utterance detection, beam search, and contextual phrase biasing.
• Backend choice: enable ONNX Runtime, LiteRT, both, neither, or implement the abstract interfaces yourself.
• Portable surface: native C++ API plus C APIs suitable for Kotlin/JNI, Swift/FFI, embedded Linux, and other hosts.
• Tested across targets: Linux, Windows, macOS, Android-oriented arm64 builds, sanitizers, and model-backed nightly lanes.

v0.0.11 highlights

• OpenAI-compatible local TTS: speech-server exposes POST /v1/audio/speech with OpenAI model aliases, native and generic voices, language and speed controls, WAV/PCM output, and optional bearer authentication.
• Windows release package: a self-contained x64 ZIP ships the server, ONNX CLI tools, speech.dll, ONNX Runtime, and a native PowerShell model downloader; CI builds and smoke-tests the extracted archive.
• DeepFilterNet3 parity: native libdf-compatible STFT scaling, ERB/complex normalization, deep filtering, overlap-add, and 480-sample delay compensation restore reference DSP behavior.
• Streaming Pocket TTS: the ONNX backend emits fixed 80 ms frames with a bounded decoder cache and an opt-in model-backed round-trip harness.
• Correct Silero v5 context: every ONNX inference now receives the graph's required 64-sample left context.

Supported models

Model	Task	ONNX	LiteRT
Silero VAD v5 · soniqo.audio	Voice activity detection	✓	✓
Parakeet TDT v3 (0.6B) · soniqo.audio	Speech-to-text	✓	✓
Whisper v3 / turbo · soniqo.audio	Multilingual speech-to-text	✓	—
Nemotron Speech Streaming (0.6B) · soniqo.audio	Streaming speech-to-text	✓	✓
Nemotron-3.5 multilingual (0.6B) · soniqo.audio	Prompt-conditioned streaming STT	✓	✓
Parakeet-EOU (120M) · soniqo.audio	Streaming STT + end-of-utterance	✓	—
Omnilingual ASR CTC (300M) · soniqo.audio	Multilingual speech-to-text	—	✓
Pyannote Segmentation 3.0 · soniqo.audio	Diarization segmentation	—	✓
WeSpeaker ResNet34-LM · soniqo.audio	Speaker embedding	—	✓
VoxCPM 0.5B	16 kHz TTS + voice cloning	✓	—
VoxCPM2 (2B) · soniqo.audio	48 kHz TTS + voice cloning	✓	✓
CosyVoice3 0.5B · soniqo.audio	24 kHz conditioned TTS	staged	—
Chatterbox · soniqo.audio	24 kHz text-to-speech	—	✓
Supertonic 3 · soniqo.audio	Text-to-speech	—	✓
Indic-Mio · soniqo.audio	Hindi/Indic voice cloning + emotion	—	✓
Kokoro 82M · soniqo.audio	Text-to-speech	✓	✓
Pocket TTS 100M	Streaming TTS (fixed Alba voice)	✓	—
DeepFilterNet3 · soniqo.audio	Speech enhancement	✓	—
Sidon · soniqo.audio	Denoise + dereverb (16 → 48 kHz)	✓	—
PersonaPlex 7B · soniqo.audio	Full-duplex speech-to-speech (CUDA)	structural	—
FunctionGemma 270M · soniqo.audio	On-device structured tool calls	—	LiteRT-LM

See docs/models.md for maturity, bundle layouts, preprocessing, memory notes, and complete examples.

Platforms and backends

Backend	Target	Platforms	Runtime setup
Core only	speech_core	Linux, Windows, macOS, Android	none
ONNX Runtime	speech_core_models	Linux, Windows, macOS, Android	extracted ONNX Runtime release via ORT_DIR
LiteRT	speech_core_models_litert	Linux x86_64, Windows x86_64, macOS arm64, Android	scripts/fetch_litert.sh / LITERT_DIR
LiteRT-LM	speech_core_models_litert_lm	macOS, Android build path	scripts/fetch_litert_lm.sh / LITERT_LM_DIR

ONNX can use CPU, Android NNAPI, Qualcomm QNN on Linux, or an application-supplied execution-provider hook. LiteRT currently uses CPU through its C API.

Quick start

Build the core and LiteRT backend:

git clone https://github.com/soniqo/speech-core.git
cd speech-core
scripts/fetch_litert.sh build/litert
cmake -B build -DCMAKE_BUILD_TYPE=Release \
    -DSPEECH_CORE_WITH_LITERT=ON \
    -DLITERT_DIR="$PWD/build/litert"
cmake --build build --parallel

Transcribe an audio buffer; Parakeet v3 detects the language automatically:

#include <speech_core/models/litert_parakeet_stt.h>

speech_core::LiteRTParakeetStt stt(
    "parakeet-encoder.tflite",
    "parakeet-decoder-joint.tflite",
    "vocab.json");

auto result = stt.transcribe(audio, sample_count, 16000);
std::cout << result.text << "\n";

Connect any implementations of the abstract VAD, STT, LLM, and TTS interfaces to the live pipeline:

speech_core::AgentConfig config;
config.mode = speech_core::AgentConfig::Mode::Pipeline;

speech_core::VoicePipeline pipeline(
    stt, tts, &llm, vad, config,
    [](const speech_core::PipelineEvent& event) {
        // transcription, response audio, tool call, or error
    });

pipeline.start();
pipeline.push_audio(mic_samples, sample_count);

Link only what your application uses:

target_link_libraries(my_app PRIVATE speech_core)
target_link_libraries(my_app PRIVATE speech_core speech_core_models)
target_link_libraries(my_app PRIVATE speech_core speech_core_models_litert)

Linux CLI packages

Releases ship .deb and .tar.gz packages for amd64 and arm64. The package bundles runtime libraries but not models.

VERSION=0.0.11
ARCH="$(dpkg --print-architecture)"   # amd64 or arm64
curl -fLO "https://github.com/soniqo/speech-core/releases/download/v${VERSION}/speech_${VERSION}_${ARCH}.deb"
sudo apt install "./speech_${VERSION}_${ARCH}.deb"

speech download-models
speech transcribe recording.wav
speech speak "Hello world" hello.wav
speech phonemize "Bonjour le monde" fr
speech serve

The amd64 package also includes the LiteRT VoxCPM2 voice-cloning command. Its x86 bundle is about 13 GB and is downloaded explicitly:

speech download-models voxcpm2
speech clone reference.wav "This is my cloned voice." cloned.wav

See the Linux CLI reference for exact syntax, model directories, standalone binaries, and the amd64/arm64 command matrix. soniqo.audio/cli documents the larger speech-swift CLI for Apple platforms.

Architecture

application audio / events
            │
            ▼
┌──────────────────────────────────────┐
│ speech_core                          │
│ VoicePipeline · turn detection       │
│ interruption · tools · audio utils   │
│ abstract VAD / STT / LLM / TTS APIs  │
└──────────────┬───────────────┬───────┘
               │               │
      ┌────────▼────────┐ ┌────▼────────────┐
      │ ONNX Runtime    │ │ LiteRT / LiteRT-LM │
      │ reference models│ │ reference models   │
      └─────────────────┘ └─────────────────────┘

The orchestration target never depends on a concrete model. A backend swap is a construction and link choice, not a pipeline rewrite.

Documentation

Topic	Documentation
Product overview and model matrix	soniqo.audio/speech-core
Linux setup	soniqo.audio/getting-started/linux
Windows setup	soniqo.audio/getting-started/windows
Linux CLI	docs/cli.md
Interfaces and custom backends	docs/interfaces.md
Model implementations	docs/models.md
Voice pipeline and state machine	docs/pipeline.md
C API / FFI	docs/c-api.md
Tool calling	docs/tools.md

Build variants

# Orchestration only: no ML runtime
cmake -B build -DCMAKE_BUILD_TYPE=Release

# ONNX models
cmake -B build-onnx -DCMAKE_BUILD_TYPE=Release \
    -DSPEECH_CORE_WITH_ONNX=ON -DORT_DIR=/path/to/onnxruntime

# LiteRT models
scripts/fetch_litert.sh build/litert
cmake -B build-litert -DCMAKE_BUILD_TYPE=Release \
    -DSPEECH_CORE_WITH_LITERT=ON -DLITERT_DIR="$PWD/build/litert"

Testing

cmake --build build --parallel
ctest --test-dir build --output-on-failure

Core tests require no model files. Backend integration tests skip cleanly unless their model-directory environment variables are set. CI covers Linux, Windows, and macOS; model-backed scheduled workflows cover the public ONNX and LiteRT bundles.

Related projects

• speech-android — Kotlin SDK and JNI integration over speech-core.
• speech-swift — native MLX/CoreML speech stack for macOS and iOS.
• Soniqo documentation — guides, architecture, benchmarks, and model pages.

Contributing

Issues and pull requests are welcome. Branch from main, build the affected configurations, run ctest, and open a focused PR.

License

Apache 2.0 — see LICENSE.