Skip to content
← Home

03 ::: failures

Failure Logs

Things that broke, what I thought first, what was actually wrong.

Engineering CV's are sanitized retellings. These are the actual logs — the wrong hypotheses, the dead ends, the single character that ate two days.

  1. log · 01 · Apr 12, 2025

    [problem] p99 latency on the RAG endpoint suddenly jumped from 1.2s to 8s overnight. p50 unchanged. No code deploys.
    [hypothesis] Postgres lock contention from a long-running migration that finished overnight.
    [steps]
    1. Pulled traces — 90% of the spike was in the embedding step, not retrieval.
    2. Embedding API itself looked healthy (vendor status green).
    3. Compared embedding latency by document length — the long tail was 50× normal.
    4. Found a customer had ingested a 600-page PDF that hit the chunking edge case: one 'paragraph' that was actually the entire document with no newlines.
    [resolution] Added a hard cap (4 KB) at the chunker before the embedding call. Re-chunked the offending docs in a backfill.
    [lesson] Tail behaviour is a function of input distribution, not just system health. Synthetic load tests use uniformly-distributed inputs and miss this.

  2. log · 02 · Feb 8, 2025

    [problem] Customer reports 'the AI is returning wrong answers about contract dates.' Eval suite all green.
    [hypothesis] Hallucination — model isn't grounding in retrieved context.
    [steps]
    1. Reproduced with the customer's exact query — got the right answer.
    2. Asked customer for a screenshot. Date was right but in MM/DD/YYYY; customer expected DD/MM/YYYY (UK).
    3. Checked our system prompt — no locale instruction.
    4. Checked the source PDF — date was actually in DD/MM. Model translated to MM/DD because that's the dominant pattern in training.
    [resolution] Added 'preserve date formatting exactly as written' to the system prompt. Added 5 UK-format date examples to the eval suite.
    [lesson] 'Wrong answer' is sometimes 'right answer in the wrong locale'. Eval suites are blind to assumptions you didn't write down.

  3. log · 03 · Nov 30, 2024

    [problem] Whisper streaming pipeline crashes ~once an hour with no clear pattern. Stack trace points to a generic GPU OOM.
    [hypothesis] Memory leak in our Python wrapper around whisper.cpp.
    [steps]
    1. Added per-stream memory tracking — usage was bounded.
    2. Profiled the GPU pool: total usage was 60% — not OOM-shaped.
    3. Inspected the kernel log: 'GPU has fallen off the bus' messages just before each crash.
    4. Reproduced by disabling ECC error correction — same failure.
    [resolution] It was hardware. The cloud provider had silently migrated us to a host with a flaky GPU. Filed a ticket; they swapped the host; the issue stopped.
    [lesson] When the trace says 'memory error' and the metrics disagree, look at the layer below. We assumed software because we wrote software. The bus didn't.