The Problem
[Acme Corp]'s legal team was drowning in document review. They needed a tool that could ingest contracts, surface risk clauses, and explain its reasoning — fast enough to be used in real meetings, accurate enough to actually trust.
The existing v1 used a single LLM call per document. It was slow (12s p95), expensive ($0.40 per query), and prone to hallucinating clause references.
My Approach
I rebuilt the pipeline around three principles:
- Retrieve before reasoning. A hybrid retriever (BM25 + dense, with re-ranking) finds the relevant passages first. The LLM only sees what it needs.
- Stream early, stream often. Speculative retrieval starts on the first keystroke; the model begins generating before retrieval finishes for tokens that don't depend on retrieved context.
- Eval-driven everything. A 400-example regression suite gates every prompt change, retriever change, and model upgrade.
Tradeoffs
I considered fine-tuning a smaller model for the routing step — would have been cheaper at scale, but added a CI footprint and ops surface I didn't think was worth it at our volume. I also skipped reranking with a cross-encoder for the first launch; the BM25 + dense hybrid was good enough to ship.
Results
- p95 latency: 12s → 180ms (66× faster, mostly streaming)
- cost per query: $0.40 → $0.06
- accuracy on the regression suite: 78% → 94%
- monthly active users: 3k → 200k in 9 months
The eval harness paid for itself in the first month — it caught two prompt regressions before they hit production that would have cost the team half a day each to debug from logs alone.