The Problem
[Beta Labs] needed live transcription for their meeting tool — sub-second latency, $0 per minute marginal cost, and the ability to run on-prem for compliance reasons. The OpenAI Whisper API was a non-starter on all three.
My Approach
A streaming pipeline around the open-source Whisper model:
- Chunked audio at 200ms intervals over a WebSocket
- Sliding-window inference with a 4-second context buffer
- Hypothesis stabilization — late chunks correct earlier output once the model has more context, but the UI shows confident output immediately
- Per-customer Redis queues so the GPU pool stays saturated without starving any one tenant
The hardest part wasn't the model — it was the audio plumbing. Browsers vary wildly in how they chunk audio, and getting this right on Safari nearly defeated me.
Tradeoffs
I went with whisper.cpp over the Python implementation because the C++ build
runs at 1/3 the memory and ~2× the throughput on the same hardware. The cost
was a more complex deployment (GLIBC, CUDA flags, the usual fun).
Results
- streaming latency: 200ms (target was sub-second)
- WER on internal benchmark: 8.2% (vs OpenAI's ~7.5% — close enough)
- monthly cost: 60% lower than the equivalent OpenAI usage
- shipped on-prem for two enterprise customers who couldn't use OpenAI