Law 30 · Evaluation & Measurement
Regress or Repeat
Every fixed bug is a future regression unless it becomes a test.
The principle
LLM systems are non-deterministic and globally coupled, so a prompt tweak that fixes one case can quietly break three others. Rerunning real production examples against a new prompt is the only way to know you didn't break what already worked. Without a regression suite you're stuck in a whack-a-mole loop, rediscovering the same failures release after release.
The mechanism, the warning signs, a worked example, and the apply-it recipe for this law are in the complete edition.
Unlock all 50 laws — $9.99Related laws
25
Averages Lie
97% overall can hide a 60% segment.
Evaluation & Measurement
26
Vibes Don't Scale
Eyeballing outputs feels like progress until you can't tell if a change helped.
Evaluation & Measurement
27
Look at Your Data
The highest-ROI activity in AI is the one teams skip first.
Evaluation & Measurement