Skip to content
← Home

02 ::: systems

System Design

Architectures I've shipped, including the boring decisions that mattered most.

The interesting parts of an LLM system are usually not the model. These are the architectures I've actually shipped — with the trade-offs that turned out to matter.

  1. system · 01

    Document-Intelligence RAG (200k MAU)

    Lawyers reviewing contracts. Sub-second responses, 94% accuracy on a regression suite of 400 cases.

    layers

    1. Next.js edge for the chat UI; streaming over SSE
    2. FastAPI on Fly.io for orchestration — 4 regions, sticky to the user's nearest
    3. Hybrid retrieval: BM25 (Postgres tsvector) + dense (pgvector) → cross-encoder rerank
    4. Anthropic Claude for the generation; Haiku for routing, Sonnet for synthesis
    5. OpenTelemetry → Honeycomb for end-to-end traces; Sentry for errors

    scale

    200k MAU · p95 180ms · ~$0.06/query

    decisions that mattered

    • Postgres + pgvector instead of a dedicated vector DB — one fewer system to operate, fast enough at our scale.
    • Reranker is a smaller cross-encoder, not the LLM. The LLM only sees 5 passages, not 50.
    • Streaming starts on the first keystroke (speculative). Most users see first token at <200ms.

  2. system · 02

    Realtime Whisper transcription pipeline

    Live meeting transcription. On-prem-deployable for compliance customers.

    layers

    1. Browser Audio API → 200ms chunks over WebSocket
    2. WebSocket gateway in Go, dispatches to a per-tenant Redis queue
    3. GPU pool running whisper.cpp; sliding 4-second context buffer per stream
    4. Hypothesis stabilization: late chunks correct earlier output, UI shows confident output immediately

    scale

    ~200ms streaming latency · 60% cheaper than the OpenAI API · 8.2% WER

    decisions that mattered

    • whisper.cpp over the Python implementation — ~2× throughput on the same GPU.
    • Per-tenant queue rather than shared — isolates noisy customers, keeps GPU pool saturated.
    • Ship the on-prem build as a Helm chart, not a container — customers want to run it in their own k8s.