Index / Work / Local Voice AI on CPU
Local Voice AI on CPU
Pipeline architecture and performance engineering
A hands-free voice assistant whose speech stack runs entirely on a CPU — no cloud speech service, no GPU required. Microphone, voice detection, transcription and synthesis all local, streamed over a WebSocket to a browser front end.
- Client
- Independent
- Industry
- Privacy-First AI
- Role
- Pipeline architecture and performance engineering
- Platform
- Web
- Stack
- FastAPI · ONNX Runtime · WebSocket · PyTorch · Python
- Dated
- 2026-07-29
- Full speech path runs locally on CPU — no cloud transcription or synthesis
- Voice-activity detection for hands-free turn taking
- ONNX Runtime backend, chosen over PyTorch on measured throughput
- Startup warm-up so the first user turn is not the slow one
- Incremental sentence splitting so speech starts before generation finishes
- Single-worker synthesis thread with reproducible seeding
- Browser front end with AudioWorklet capture, no build step
① Challenge
Local speech stacks are usually either too slow to hold a conversation or too heavy to run without a GPU. Stacking voice detection, transcription and synthesis in one process also creates conflicts that never show up in isolation — several of these libraries quietly mutate global thread settings when they load, and mixing GPU runtime versions crashes the process outright.
② Solution
The pipeline is built around a sub-4M-parameter open-weight speech synthesis model (Apache-2.0) served through ONNX Runtime, which measured consistently faster and more stable than the PyTorch path. Synthesis is pinned to CPU deliberately: the transcription library bundles a different GPU runtime version, and mixing the two crashes the process. Model warm-up runs at startup rather than on the first user turn, because the first several calls run below real time before settling. Thread counts are re-asserted defensively after each library loads.
③ Result
A working hands-free conversation loop on ordinary CPU hardware, with the entire speech path — transcription and synthesis — running locally. Only text ever leaves the machine. This is the same speech layer that makes on-premise deployment possible in the voice platform work.