Law 28 · Evaluation & Measurement
The Judge Is Biased
An LLM grader reacts to length and position, not just substance.
The principle
An LLM judge can match human preferences over 80% of the time, but only after you account for its systematic biases: position bias (favoring the first answer shown), verbosity bias (favoring longer answers regardless of quality), and self-enhancement bias (favoring its own outputs). It's a useful instrument, but an uncalibrated one that grades surface features as readily as substance.
The mechanism, the warning signs, a worked example, and the apply-it recipe for this law are in the complete edition.
Unlock all 50 laws — $9.99Related laws
25
Averages Lie
97% overall can hide a 60% segment.
Evaluation & Measurement
26
Vibes Don't Scale
Eyeballing outputs feels like progress until you can't tell if a change helped.
Evaluation & Measurement
27
Look at Your Data
The highest-ROI activity in AI is the one teams skip first.
Evaluation & Measurement