The Problem
Most LLM eval libraries are batch tools — slow, opinionated, and built around "score this dataset" workflows. They're great for research and useless for the iteration loop you actually want during development: edit the prompt, hit save, see what broke.
I wanted pytest for agents.
My Approach
A minimal framework where each eval is a Python function decorated with
expectations. Runs in parallel, caches model calls aggressively, integrates
with pytest -k so you can run a single eval in under a second.
@eval_case(rubric="must mention notice period")
async def test_freelance_question(agent):
response = await agent.ask("Are you available for freelance?")
assert_contains(response, ["notice", "Q3"])The killer feature turned out to be deterministic replay: every model response is captured to a fixture file, so re-running the same eval suite is free and reproducible. Real model calls only happen when you explicitly re-record.
Tradeoffs
I deliberately punted on rubric-based judging (LLM-as-judge) for v1 — the non-determinism would have undermined the "fast feedback loop" promise. v2 adds it as an opt-in.
Results
- 1.2k GitHub stars in 6 months
- adopted by 4 production teams I know of (probably more)
- caught 12 regressions in our internal pipelines before they shipped
- 4× faster dev loop than our previous batch-based eval setup