Law 47 · Trust & Coordination
Trust Is Calibrated, Not Granted
Autonomy is earned in proportion to track record.
The principle
People give an agent freedom the way they give it to a new hire: a little at a time, on reversible things first, loosening the leash only as it proves itself. Both failure modes are real. Over-trust leads to misuse, under-trust leads to a good capability being abandoned. Reliance follows the reliability a system appears to have, not just the reliability it actually has.
Why it happens
Good trust is calibrated trust: people rely on the agent where it is reliable and hold back where it is weak. Over-trust causes misuse; under-trust causes disuse. Both waste capability. Algorithm-aversion research shows people can abandon a useful system after seeing one error, even when they would forgive the same error from a human. The answer is not hype or concealment. Show the track record, expose uncertainty, and widen autonomy gradually. Trust should follow demonstrated competence, not marketing, novelty, or fear.
Watch for
- The agent is given broad write access to high-stakes systems before it has a track record on reversible ones.
- Every single action is funneled through manual approval, and the team is quietly abandoning the tool from fatigue.
- The agent presents strong and shaky outputs with identical confidence, giving users no basis to calibrate.
In practice
Two failure modes, both expensive. On day one you give the agent direct write access to production billing and it confidently double-applies a discount rule across 800 accounts. Or, burned by that, you wire every single action through manual approval, the team drowns in confirmation fatigue, and within a month they have quietly stopped using a genuinely capable tool. Calibrate instead of swinging between extremes: start it on reversible, low-stakes actions, widen the leash as its track record proves out, and surface where it is reliable versus where it is guessing so people lean on it exactly where they should and not an inch further.
Apply it
- Start the agent on low-stakes, reversible actions and widen its blast radius only as reliability is proven.
- Surface where the agent is reliable versus where it is guessing so users rely on it exactly that far.
- Avoid both extremes: neither hand it production write access on day one nor gate every trivial action behind approval.
The takeaway
Start the agent on low-stakes, reversible actions and widen its blast radius as it proves reliable. Show why it's confident where it's strong and flag where it's weak, so people lean on it exactly where they should.