Law 27 · Evaluation & Measurement
Look at Your Data
The highest-ROI activity in AI is the one teams skip first.
The principle
Error analysis, reading your app's actual traces by hand to find where it fails, is the single most valuable thing you can do when building with AI, yet teams skip it for dashboards and vanity metrics that climb while users still struggle. You can't write a good eval for a failure mode you've never seen, and you only see failure modes by reading transcripts.
The mechanism, the warning signs, a worked example, and the apply-it recipe for this law are in the complete edition.
Unlock all 50 laws — $9.99Related laws
25
Averages Lie
97% overall can hide a 60% segment.
Evaluation & Measurement
26
Vibes Don't Scale
Eyeballing outputs feels like progress until you can't tell if a change helped.
Evaluation & Measurement
28
The Judge Is Biased
An LLM grader reacts to length and position, not just substance.
Evaluation & Measurement