01 ::: thinking
How I Think
The reasoning engine. How I decompose problems before writing code.
Most engineering problems aren't bottlenecked by what you know — they're bottlenecked by the order in which you consider what you know. These are decision walks I run before committing to an approach.
branch · 01
User reports the LLM feature is 'sometimes hallucinating clause references' in production.
decompose
- Is the retriever pulling the right passages?
- Is the model ignoring retrieved context and confabulating?
- Is the prompt pinning the model to ground in retrieved text?
- Is this happening on a specific document type, or across the board?
approach
Build a regression suite of 30 known-bad outputs first. Then bisect: pin the prompt, swap retrievers; pin the retriever, swap prompts. The eval suite is the only reliable signal — log inspection alone misleads.
rejected
- 'Just upgrade to a smarter model' — masks the underlying retriever bug and balloons cost.
- 'Add a self-critique step' — bandaid that adds latency without fixing the source.
outcome
Found the retriever was returning low-relevance passages on a specific layout (two-column PDFs). Reranker fix dropped hallucination rate by 80%. Prompt + model unchanged.
branch · 02
Pipeline p95 is 4× our SLO. Where does the budget go?
decompose
- What's the actual SLO and what's the current p95?
- Single-shot or chained calls? If chained, which stage dominates?
- Is it the model, the network, or our own code?
- Is the spike on the long tail (specific query types) or uniform?
approach
Instrument every stage with request-level traces before optimising anything. The intuition about where time goes is almost always wrong.
rejected
- 'Switch to streaming' — improves perceived latency but doesn't fix the actual budget.
- 'Cache aggressively' — easy win on hot paths but doesn't help cold queries that drive p95.
outcome
Reranker stage was 60% of latency. Replaced with a faster cross-encoder on a smaller candidate set. p95 fell to 1.2× SLO; the rest came from removing a redundant validation hop.
branch · 03
Should we fine-tune for this task or stick with prompting?
decompose
- Do we have ≥1k high-quality labeled examples?
- Is the task narrow enough that fine-tuning won't degrade general behaviour we depend on?
- What's the cost delta — both training and inference?
- What's the operational cost of maintaining a custom model?
approach
Default to prompting + few-shot until prompting demonstrably can't solve it. Fine-tuning is a one-way door: you now own model ops, eval, drift, and retraining cadence.
rejected
- 'Fine-tune for the prestige factor' — unfortunately a real driver in many teams.
- 'Fine-tune to save inference cost' — usually false economy once you account for training and re-evaluation.
outcome
Prompting + few-shot covered 94% of cases. The remaining 6% routed to a smaller fine-tuned classifier. Avoided owning a generative model.