Law 43 · Humans & Autonomy
Automation Bias
People will trust the machine over their own eyes.
The principle
Give people an automated aid and they make errors of omission (missing problems it didn't flag) and commission (following its recommendation even when their own valid evidence says otherwise). The automation becomes a shortcut that replaces careful checking, so the agent's recommendation doesn't just inform the human. It overrides their independent judgment.
Why it happens
Automation bias is what happens when the machine's verdict replaces independent checking. People miss problems the system did not flag, and they follow recommendations even when other evidence disagrees. The driver is cognitive economy: verifying takes effort; accepting the recommendation is easy. High reliability can make the bias stronger because each correct call trains people to stop looking. The interface matters. If it shows only the conclusion, it invites rubber-stamping. Show the evidence beside the verdict and make disagreement as easy as agreement.
Watch for
- Reviewers approve the agent's recommendation at near-100% rates, far faster than it would take to actually inspect the evidence.
- The interface shows the verdict prominently but buries or omits the raw signals the verdict was based on.
- Disagreeing with the agent takes more clicks or justification than agreeing with it.
In practice
Your fraud-review agent flags a transaction as low risk, auto-approve and presents that verdict as a single green badge. The analyst clicks approve without opening the underlying signals, even though the shipping address changed three minutes after a password reset, a pattern they would have caught in a heartbeat on their own. If the recommendation is the only thing on screen, you have built a rubber-stamp machine, not a decision aid. Put the raw evidence next to the verdict, make 'I disagree' a one-click action with no friction, and occasionally withhold the recommendation entirely to keep the human actually looking.
Apply it
- Present the raw evidence next to the recommendation, never the verdict alone.
- Make disagreement a frictionless, one-step action that needs no special justification.
- Periodically withhold the recommendation entirely so the human has to form an independent judgment.
The takeaway
Never present an agent's output as the only signal. Make the human look at the raw evidence next to the recommendation, and make it cheap to disagree.