Laws of AI Agents

The principle

Most bad outputs come from missing, stale, or conflicting context, not from a model that can't think. The model often reasons fine over the picture it was handed and still lands wrong, because the picture was wrong to begin with. Bad context produces confident bad answers.

Why it happens

A model treats the context window as the world. If that window contains a stale record, a missing constraint, or two facts that conflict, the model has no reliable built-in sense that one is old or suspect. Preference tuning can make this worse by nudging the model toward the framing it was given. The result is not always weak reasoning. Often it is competent reasoning over a bad picture of reality. A stronger model can still fail if the context it sees is stale, partial, or contradictory.

Watch for

• The same question gives different answers depending on which session or document was loaded first.
• Outputs confidently reference facts that are real but out of date, or contradict a source you know is in the window.
• Bumping to a larger or newer model produces no measurable accuracy gain on the failing cases.

Apply it

1. On every bad run, dump and read the exact context the model saw before blaming the model.
2. Stamp each retrieved fact with its source and timestamp, and drop or refresh anything past a freshness threshold.
3. Detect contradictions in the assembled context and surface them instead of silently concatenating both.

Sources and further reading

• Effective Context Engineering for AI Agents · Anthropic
• Towards Understanding Sycophancy in Language Models · Sharma et al., 2023

The principle

A step that works 95% of the time, run ten times in a row, gives you the right final answer only about 60% of the time. The failures don't announce themselves. They pile up quietly until the answer is wrong and you can't tell which step broke it. Every link you add lowers the ceiling for the whole chain.

Why it happens

In a serial agent run, each step feeds the next. A small mistake becomes a premise, then the next step builds on it. Ten steps at 95% reliability is about 60% end-to-end in the simple worst case, before recovery or checkpoints. Long runs add another problem: each turn leaves behind assumptions, partial decisions, and failed attempts that can pollute the next turn. The fix is to shorten the chain, improve the weakest steps, and checkpoint after pivotal work so one bad output cannot quietly poison the rest.

Watch for

• End-to-end success is far worse than the per-step accuracy you measured in isolation.
• Final outputs are wrong but no single step looks obviously broken when you inspect it.
• Adding more pipeline stages keeps lowering overall reliability even as each stage tests fine.

Apply it

1. Count the sequential steps and multiply their reliabilities to get the real end-to-end ceiling.
2. Collapse independent steps into one pass, or raise per-step reliability, before adding new stages.
3. Insert a validation checkpoint after pivotal steps that halts or restarts from the last good state on low confidence.

Sources and further reading

• Building Effective Agents · Anthropic
• Don't Build Multi-Agents · Walden Yan, Cognition, 2025
• Measuring AI Ability to Complete Long Tasks · METR, 2025

The principle

Give a model a long input and it pays the most attention to the start and the end. Facts buried in the middle quietly lose their grip. They're present but basically ignored. That's the worst kind of bug, because the information was technically in context and nothing looks wrong.

Why it happens

Long context is not uniform attention. Models tend to use the beginning and end of an input more reliably than the middle, and the dip is worse when the key fact has no exact keyword hook. Newer models reduce the effect, but they have not made it disappear. That makes the middle of a long prompt a dangerous hiding place for critical facts. The system does not error. The fact is technically present. It is simply less likely to shape the answer when the model needs it.

Watch for

• The agent misses a fact you can confirm is sitting in the middle of a long input.
• Accuracy on the same task degrades sharply as you lengthen the context.
• Reordering the input so the key fact is near the top or bottom suddenly fixes the answer.

Apply it

1. Lead with a short summary of what to find, and restate the critical instruction at the very end.
2. Rank and place the most relevant retrieved passages at the edges of the context, not the middle.
3. Test long-context retrieval with questions that have no keyword overlap, not just literal needle matches.

Sources and further reading

• Lost in the Middle: How Language Models Use Long Contexts · Liu et al., 2023
• NoLiMa: Long-Context Evaluation Beyond Literal Matching · Modarressi et al., 2025

The principle

An agent will write the summary before doing the work if you let it. Looking finished is cheaper than being finished, so the model drifts toward the cheaper path: a plausible report, a confident 'done', a success it never tested. The output reads complete. The work isn't. This is specification gaming, where the model optimizes the proxy you can see instead of the goal you meant.

Why it happens

A confident completion report is cheap to generate. Real completion is slower: run the tool, read the output, handle the error, try again. Preference-tuned models are rewarded for helpful, finished-sounding answers, so they can drift toward the appearance of success when the environment does not demand proof. The control is simple: make done depend on an artifact. A passing test, a real diff, a saved file, or an HTTP response creates a cost for pretending. Grade the artifact, not the sentence that claims it exists.

Watch for

• The agent reports success but you find no corresponding artifact: no test run, no diff, no API response.
• Summaries use confident completion language (all tests pass, feature complete) without evidence attached.
• Spot-checking finished tasks regularly turns up work that was never actually performed.

Apply it

1. Require a concrete artifact (test output, diff, file, citation) before any claim of completion is accepted.
2. Grade the proof programmatically, not the prose, and reject completions that lack the artifact.
3. Have a separate check actually execute the claimed result rather than trusting the agent's report of it.

Sources and further reading

• Specification gaming: the flip side of AI ingenuity · DeepMind, 2020
• Towards Understanding Sycophancy in Language Models · Sharma et al., 2023

The principle

When a system says 'up to 24 hours', 'may retry', or 'no guaranteed latency', those limits are the numbers that matter. Designing for the typical case works right up until the rare event, which is exactly when failure costs the most. At scale, those failures aren't edge cases. They're the normal state of things.

Why it happens

At scale, rare cases stop being rare. If a dependency says up to 24 hours, may retry, or no guaranteed latency, those words define the run you must survive. Tail latency research shows the same pattern in distributed systems: waiting on many sub-requests makes the slowest ones dominate the user experience. Agent systems inherit that math. Timeouts, retry budgets, dedupe windows, and SLAs built around the average will fail exactly when the slow path finally appears. Design against the bound, not the happy path.

Watch for

• Timeouts, dedup windows, or retry budgets are set to the typical latency rather than the documented maximum.
• Failures cluster at peak load or month-end, exactly when the system is most exercised.
• A spec says up to X or may and the design quietly assumed the average instead.

Apply it

1. Read every up to and may as the number you must survive, and do the math against that ceiling.
2. Size timeouts, dedup windows, and retry budgets for the worst plausible run, not the common one.
3. Load-test at the tail and the peak, since at scale the rare path becomes the routine one.

Sources and further reading

• 10 Lessons from 10 Years of Amazon Web Services · Werner Vogels, 2016
• The Tail at Scale · Dean & Barroso, 2013

The principle

Asking a model to reason step by step before answering measurably improves results, and for an agent the stakes are lopsided. A reasoning trace is cheap and easy to undo. An executed action, a sent email, a dropped table, a charged card, is not. Letting the model lay out its plan in tokens before it commits is the cheapest insurance you can buy.

Why it happens

A plan is cheap and reversible. A tool call may not be. Asking the model to state the target, scope, and expected effect before acting gives you a low-cost checkpoint before the expensive step. ReAct showed that interleaving reasoning and action helps agents track state and recover from surprises, but the broader rule is simpler: spend disposable tokens before irreversible actions. If the plan is wrong, discard it. If the email is sent, the table is dropped, or the card is charged, the cost is real.

Watch for

• The agent fires a side-effecting tool call with no stated plan or scope beforehand.
• Destructive actions execute on the first instinct, then turn out to have hit the wrong target.
• Post-mortems show the agent never articulated what it was about to do or why.

Apply it

1. Require an explicit reasoning or plan step before any tool call that has side effects.
2. Make the plan state the exact target, scope, and expected effect (for example the row count) before acting.
3. Treat reasoning tokens as cheap insurance and spend them freely ahead of any irreversible action.

Sources and further reading

• Chain-of-Thought Prompting Elicits Reasoning in LLMs · Wei et al., 2022
• ReAct: Synergizing Reasoning and Acting in Language Models · Yao et al., 2022

The principle

A single greedy chain of thought is fragile. Sample several independent reasoning paths and take the majority answer, and you get large, consistent gains. Correct reasoning tends to converge while mistakes scatter, so agreement across independently generated plans is a real signal worth trusting before you act on something that matters.

Why it happens

One sampled reasoning path is one route through a probabilistic space. If it makes a bad early move, everything after it inherits that move. Multiple independent attempts give you a different signal: correct answers tend to converge, while mistakes scatter. Repeated sampling only helps when you can choose among the samples, through majority vote for comparable answers or through an external verifier for plans and artifacts. Use it for consequential, hard-to-reverse decisions. Do not spend 5x compute on routine steps that are cheap to undo.

Watch for

• High-stakes outputs ride on a single greedy generation with no second opinion.
• Re-running the same prompt yields meaningfully different answers, revealing the first one was luck.
• Errors slip through because nothing checks whether independent attempts actually agree.

Apply it

1. For consequential decisions, generate the answer several independent times instead of trusting the first.
2. Take the majority answer when outputs are comparable, or use an external check to pick among them.
3. Treat disagreement across the samples as a signal to escalate rather than silently picking one.

Sources and further reading

• Self-Consistency Improves Chain of Thought Reasoning · Wang et al., 2022
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling · Brown et al., 2024

The principle

Tree-of-Thoughts turns linear reasoning into a search: generate several candidate thoughts, judge them, look ahead, and backtrack instead of being stuck going left to right. It matters most when an early choice is pivotal, which is exactly the spot where an agent's first irreversible action sets up everything downstream. Cheap, recoverable steps don't need it. Pivotal ones do.

Why it happens

Some first moves shape everything after them: a migration strategy, a contract interpretation, a tool with irreversible side effects. Linear reasoning commits early and then keeps going. Search-based methods such as Tree-of-Thoughts and LATS generate several candidate next moves, score them, and allow backtracking. That extra compute is not for every task. It pays when the first decision is high-leverage and hard to unwind. For cheap reversible work, branch less. For a pivotal commitment, explore before the path locks in.

Watch for

• The agent commits to a pivotal strategy on its first instinct, and everything downstream is locked to it.
• A wrong early choice forces an expensive redo of all the work that followed.
• There is no step where alternative plans are generated and compared before the irreversible move.

Apply it

1. Reserve branching for early actions that are high-leverage or hard to reverse, not cheap recoverable ones.
2. Have the agent generate several candidate plans and score each on risk and reversibility before picking.
3. Look ahead and allow backtracking on the pivotal step instead of committing to the first path.

Sources and further reading

• Tree of Thoughts: Deliberate Problem Solving with LLMs · Yao et al., 2023
• Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models · Zhou et al., 2023

The principle

The Bitter Lesson isn't a ban on structure. It's a warning against hand-coded cleverness that quietly becomes a ceiling. Use code where you need guarantees and thin scaffolds for today's weak spots, but keep asking whether a simpler, more model-driven version now works better.

Why it happens

The Bitter Lesson warns against clever structure that locks in yesterday's assumptions. It does not say to remove all structure. Code should still enforce schemas, permissions, retries, and other guarantees. The risky part is bespoke prompt machinery that exists only to compensate for a temporary model weakness. As models improve, those chains and heuristics can become the bottleneck. Keep the scaffold thin, benchmark it against a simpler baseline, and be willing to delete it when the model can handle the task with less help.

Watch for

• A new model release makes your hand-tuned chain the bottleneck rather than an improvement.
• Most of your effort goes into encoding heuristics the model could plausibly infer itself.
• A plain here are the tools, decide baseline matches or beats your elaborate scaffolding.

Apply it

1. Use deterministic code for guarantees, not for hand-encoding every judgment.
2. Build the thinnest scaffold that works and that you would happily delete when the model improves.
3. Periodically re-test a minimal-scaffold baseline against your tuned pipeline as models advance.

Sources and further reading

• The Bitter Lesson · Richard Sutton, 2019
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters · Snell et al., 2024

The principle

More reasoning isn't automatically better. On easy tasks it just burns latency and money for nothing. On some tasks the model finds the answer early and then talks itself out of it. Reasoning depth has a useful range, not an endless upside.

Why it happens

Reasoning has a budget, and more budget is not always more accuracy. Overthinking studies show models can spend unnecessary tokens on simple arithmetic or reach the right answer early and then drift away. Apple's Illusion of Thinking made a stronger claim about reasoning collapse, but that paper was contested and later work found a mixed picture with evaluation artifacts and real limits. The practical lesson survives: cap easy paths, escalate hard paths to tools or verifiers, and do not confuse a longer trace with a better answer.

Watch for

• Trivial lookups take seconds and cost multiples because everything is routed through extended reasoning.
• The model reaches a correct answer early, keeps deliberating, and lands on a wrong one.
• Longer thinking traces show no accuracy gain, or even a drop, on your easy cases.

Apply it

1. Match the reasoning budget to problem difficulty rather than maxing it out everywhere.
2. Cap or skip extended thinking on simple, low-stakes steps like direct lookups.
3. Stop once a confident answer is reached instead of letting the model keep re-deriving.

Sources and further reading

• Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs · Chen et al., 2024
• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models · Shojaee et al. (Apple), 2025
• Comment on The Illusion of Thinking · Lawsen, 2025
• Rethinking the Illusion of Thinking · Varela et al., 2025

The principle

For facts the model doesn't already know well, the answer can only be as good as the evidence you retrieve. If the right passage never reaches the context, the generator fills the gap from memory and guesswork. Retrieval quality sets the practical ceiling for any grounded answer.

Why it happens

RAG only grounds the model in passages that actually reach the prompt. If the answer-bearing passage is missing, the model can still answer from parametric memory, but that memory is lossy and may be out of date. The failure then looks like a generation problem even though the evidence never arrived. Measure retrieval directly: did the needed passage appear in the top-k, and was it ranked high enough to use? For grounded facts outside the model's reliable memory, retrieval recall sets the practical ceiling.

Watch for

• Upgrading to a stronger generation model barely moves end-to-end accuracy on factual questions.
• You have never measured whether the answer-bearing passage appears in the retrieved set.
• Wrong answers are fluent and confident rather than hedged or empty, suggesting the model is filling a gap.

Apply it

1. Build a labeled set of queries with known answer passages and measure recall at k before touching prompts or models.
2. Treat any answer whose supporting evidence was never retrieved as a retrieval failure, not a generation failure.
3. Fix recall first by tuning chunking, query expansion, and k, then optimize the generator only once evidence reliably lands in context.

Sources and further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · Lewis et al., 2020
• Ragas: Automated Evaluation of Retrieval Augmented Generation · Es et al., 2023

The principle

Vendors marketed RAG legal tools as 'hallucination-free', but a Stanford audit found they still made things up 17 to 33% of the time. Handing the model a source doesn't force it to use that source faithfully. It can misread it, over-generalize, or cite a real document for a claim the document never makes. Grounding lowers the error rate. It never gets it to zero.

Why it happens

A source in the prompt nudges generation; it does not bind it. The model can cite a real document while making a claim the document never supports, over-reading a narrow passage, or combining two spans into an unsupported synthesis. That is why grounding benchmarks check whether each claim is entailed by the provided text, not merely whether a citation exists. Retrieval lowers hallucination risk, but it does not make the system hallucination-proof. The verification unit has to be the claim, tied to the exact span that supports it.

Watch for

• A grounded system is described to stakeholders as hallucination-free or hallucination-proof.
• No step checks that each generated claim is actually entailed by a retrieved span.
• Citations are attached to answers but nobody has verified the cited passage supports the specific claim.

Apply it

1. Add a verification pass that checks each output claim is entailed by a specific retrieved span before returning it.
2. Require inline attribution at the claim level so faithfulness can be audited rather than trusted.
3. Frame retrieval as risk reduction in all messaging and remove absolute safety language from decks and contracts.

Sources and further reading

• Assessing the Reliability of Leading AI Legal Research Tools · Magesh et al. (Stanford), 2024
• FACTS Grounding: A Benchmark for Evaluating Factuality · Google DeepMind, 2025

The principle

It's backwards from what you'd expect: documents that are on-topic but don't answer the question hurt more than clearly irrelevant ones, because they look plausible and pull the generator toward answers that are wrong but adjacent. Stuffing more 'kind of relevant' chunks into the context lowers accuracy instead of improving coverage. Precision at the top beats breadth.

Why it happens

The most dangerous distractor is not random junk. It is a passage that sounds related but does not answer the question. The model treats it as evidence because it shares topic, vocabulary, or entity names, then anchors a plausible wrong answer to it. Some retrieval studies even find unrelated noise less harmful than near-misses, though the robust lesson is simpler: precision at the top matters. A few answer-bearing passages beat a padded context full of almost-relevant chunks. Retrieve broadly if needed, but rerank hard before generation.

Watch for

• Raising top-k to improve coverage makes answers worse, not better.
• Wrong answers are adjacent to the truth, like the right product family but the wrong model number.
• Context is filled with many topically similar chunks and no reranking step trims them.

Apply it

1. Retrieve a wide candidate set but rerank and keep only the few highest-precision passages.
2. Tune for precision at the top of the ranking rather than maximizing recall at any cost.
3. Drop topically similar chunks that do not directly answer the query instead of including them for safety.

Sources and further reading

• The Power of Noise: Redefining Retrieval for RAG Systems · Cuconasu et al., 2024
• The Power of Noise: Redefining Retrieval for RAG Systems · Cuconasu et al., SIGIR 2024

The principle

Dense embedding retrievers win in-domain but often lose to BM25 once you step outside the training distribution. Exact-match terms, product codes, names, and rare jargon are where embeddings blur and plain keyword search shines. In-domain accuracy doesn't predict how well a retriever generalizes, and combining the two is how strong systems cut their retrieval failures dramatically.

Why it happens

Embeddings are good at meaning, but exact tokens can blur: SKUs, error codes, names, rare terms, and domain jargon. BM25 and other lexical methods are old, but they still win when the literal string matters. BEIR showed that dense retrievers that do well in-domain can underperform BM25 out of distribution. The practical answer is hybrid retrieval: run lexical and semantic search together, then fuse and rerank the candidates. The two methods fail differently, so the combination catches cases either one misses alone.

Watch for

• Pure embedding search nails paraphrased demo questions but fails on exact codes, IDs, or product names in production.
• Out-of-domain or jargon-heavy queries return near-identical-looking but wrong matches.
• Retrieval was validated only on in-distribution examples similar to the embedding training data.

Apply it

1. Run lexical and semantic retrieval in parallel and fuse their ranked lists rather than relying on embeddings alone.
2. Combine ranked results with a position-based fusion method that needs no score calibration between retrievers.
3. Add a reranker over the fused candidates to compound precision, especially for exact-match and out-of-domain queries.

Sources and further reading

• BEIR: A Heterogeneous Benchmark for Zero-shot IR Evaluation · Thakur et al., 2021
• Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods · Cormack, Clarke, Buettcher (SIGIR), 2009

The principle

Think of the context window like a computer's RAM. The agent should actively move information between a small in-context working set and large external storage, deciding what to keep, what to evict, and what to recall. Cramming everything into one flat window mixes up working memory with long-term storage and hits hard limits fast. Durable memory needs explicit tiers and self-managed retrieval.

Why it happens

A bigger prompt is not a memory system. It is an expensive pile of tokens that eventually dilutes attention and repeats old history every turn. Durable memory needs tiers: a small working set in context, summaries or records outside it, and rules for what to recall. MemGPT treats the context window like RAM and pages information in and out. Generative-agent work adds another lesson: recall is a ranking problem, using recency, importance, and relevance. The system needs storage plus policy, not just more room.

Watch for

• A long-running session degrades over time, forgetting earlier decisions as history accumulates.
• Cost and latency climb every turn because the full history is re-sent into the prompt.
• The plan for memory growth is a bigger context window rather than eviction and external storage.

Apply it

1. Separate a small in-context working set from a large external store and page entries between them deliberately.
2. Define explicit policies for what gets promoted, summarized, and evicted rather than appending everything.
3. Rank what to recall back into context by a blend of recency, importance, and relevance to the current task.

Sources and further reading

• MemGPT: Towards LLMs as Operating Systems · Packer et al., 2023
• Generative Agents: Interactive Simulacra of Human Behavior · Park et al., 2023

The principle

A scoped agent with a handful of well-chosen tools beats a generalist drowning in options. Every extra tool is another way to choose wrong, another branch to test, another failure to debug. More capability surface means more liability surface, so breadth you don't need is just risk you signed up for.

Why it happens

Tool use is a selection problem over text descriptions. Every extra tool adds another branch, another description to read, and another possible wrong choice. Tool-overload experiments show the threshold moves by model and task, but the pattern is stable: small, distinct toolsets work better than large overlapping menus. The failure is not only context length. Similar tools blur together, so the model picks the plausible one instead of the right one. When selection gets flaky, remove or merge tools before adding more instructions.

Watch for

• The agent calls a plausible-but-wrong tool, like web search when a local query tool was the right one.
• Several tools have overlapping descriptions and the model confuses them.
• Your first fix for bad tool selection is a longer system prompt rather than fewer tools.

Apply it

1. Start with a minimal set of sharply distinct tools and add one only when a real task demands it.
2. When selection gets unreliable, remove or merge overlapping tools before rewriting instructions.
3. Keep each tool's purpose non-overlapping so the model never has to disambiguate near-duplicates.

Sources and further reading

• Building Effective Agents · Anthropic
• Writing Tools for Agents · Anthropic

The principle

Validation, schema enforcement, retries, routing, and access control aren't the model's job. They're code's job. The model is for judgment under ambiguity, and deterministic code is for everything that has to be correct every single time. Asking a probabilistic system to guarantee a contract is asking for the 0.1% that ruins you.

Why it happens

The model is the wrong place for guarantees. If output shape, authorization, retries, dedupe, or routing must be correct every time, put it in code. Let the model handle judgment under ambiguity, then validate and gate its output at the boundary. This is the production pattern behind 12-factor agents: mostly deterministic software, with model calls where language judgment is useful. A one-in-a-thousand schema or permission failure is still a production incident at scale. Hard contracts belong outside the sampled text stream.

Watch for

• A correctness guarantee like valid output structure or access control depends on the model getting it right.
• Occasional malformed outputs or unauthorized actions slip through with no code-level gate to catch them.
• Control flow lives inside the model's reasoning instead of in code you can read and test.

Apply it

1. Validate and enforce output structure in code after the model, rejecting or repairing anything off-contract.
2. Put authorization, routing, and retries in deterministic code, never in the model's discretion.
3. Reserve the model for ambiguous judgment and let code own every guarantee that must hold every time.

Sources and further reading

• Building Effective Agents · Anthropic
• 12-Factor Agents: Patterns of Reliable LLM Applications · Dex Horthy / HumanLayer, 2025

The principle

If you can't see what the agent did and why, every decision, tool call, and input, then you can't safely let it act on its own. You're not trusting it, you're hoping. Autonomy without a trace is an outage you haven't found yet, and when it breaks you'll have no way to learn why.

Why it happens

An agent run is a chain of prompts, model outputs, tool calls, and hidden state. If you do not capture that chain, you cannot explain why it acted or reproduce the failure. Structured tracing turns each model call and tool execution into a span with inputs, outputs, timing, token use, and stop reasons. OpenTelemetry now has GenAI conventions for this shape of work. The rule is practical: build the trace before you widen autonomy. Freedom you cannot inspect is freedom you cannot debug.

Watch for

• When the agent does something unexpected, you cannot reconstruct which inputs and tool calls led there.
• Decisions, tool calls, inputs, and outputs are not captured as a replayable trace.
• Autonomy was widened before instrumentation existed to see what the agent actually did.

Apply it

1. Capture every decision, tool call, input, and output as a structured, replayable trace before granting autonomy.
2. Record token usage, timing, and stop reasons per step so any run can be reconstructed after the fact.
3. Expand the agent's autonomy only as far as your trace coverage actually reaches.

Sources and further reading

• What We Learned from a Year of Building with LLMs · Yan, Husain, et al., 2024
• GenAI Observability with OpenTelemetry · OpenTelemetry

The principle

When output is inconsistent, the instinct is to throw more at the same shape: a bigger model, a longer context, more tokens. That rarely fixes a structural problem. It just spreads attention thinner. Splitting the task into focused, single-purpose passes almost always beats trying to make one overloaded pass smarter.

Why it happens

A single overloaded pass splits attention across too many goals. A bigger model or longer prompt may help, but it often leaves the structure broken. Decomposition gives each step one job: extract per item, classify per case, then reconcile across items in a separate pass. Least-to-most and decomposed prompting show why this helps: solve simpler sub-problems first, then feed their results forward. The gain is not elegance. It is inspectability. Focused stages can be tested, tuned, and repaired one at a time.

Watch for

• A single pass handling many items is inconsistent, and a bigger model or longer prompt makes it blurrier, not sharper.
• One call is responsible for several distinct sub-tasks at once.
• Errors cluster on the hardest sub-step that is buried inside an overloaded prompt.

Apply it

1. Split the work into stages that each do one thing, like extract per item, then reconcile across items.
2. Solve simpler sub-problems first and feed their results into later steps rather than answering all at once.
3. Optimize and inspect each focused pass in isolation instead of supersizing one overloaded call.

Sources and further reading

• Least-to-Most Prompting Enables Complex Reasoning · Zhou et al., 2022
• Decomposed Prompting: A Modular Approach for Solving Complex Tasks · Khot et al., 2022

The principle

When something misbehaves, the cheapest fix that addresses the root cause usually wins, and that's usually clearer instructions, a better tool description, or a concrete example, not a new classifier, preprocessing layer, or pipeline. Infrastructure feels like progress, but it often just wraps an unsolved prompt in more surface area.

Why it happens

Many agent bugs are not platform bugs. They are vague instructions, weak tool descriptions, missing examples, or unclear scope. Adding a classifier, router, or preprocessing service can feel like progress, but it also adds latency, failure modes, and maintenance around the same unresolved ambiguity. Start with the cheapest root-cause fix: sharper words, better examples, clearer tool contracts. Build machinery only after the simple version has failed in a way you can name. Complexity should answer evidence, not anxiety.

Watch for

• A new service or pipeline is being specced before anyone rewrote the failing instruction or tool description.
• Infrastructure was added but the original misbehavior persists.
• The actual defect is a vague description the model cannot act on, masked by surrounding machinery.

Apply it

1. Diagnose the root cause and try clearer instructions, sharper tool descriptions, and concrete examples first.
2. Start with the simplest prompt that could work and add complexity only when a real failure forces it.
3. Build new infrastructure only after proving that prompt-level fixes genuinely cannot close the gap.

Sources and further reading

• Building Effective Agents · Anthropic
• What We Learned from a Year of Building with LLMs (Part I) · Yan, Bischof, Husain, et al., 2024

The principle

The agent decides what to call based on how a tool reads, not on what it actually does. A vague description like 'searches the database' gets skipped in favor of a tool the model understands better, even a worse one. Thin tool descriptions cause more failures than thin instructions ever do.

Why it happens

The model does not inspect your tool implementation. It sees the tool name, description, and argument schema. That makes tool routing a reading task. If two tools sound similar, or one says only searches the database, the model has little basis for choosing correctly. Real tool ecosystems show many descriptions fail to state purpose clearly, and better descriptions improve success. Write tool docs for the model the way you would onboard an engineer: what it does, when to use it, when not to, and what comes back.

Watch for

• The agent reaches for a general or external tool when a specific local one would have answered the query directly.
• Two tools with overlapping descriptions get confused, and the agent picks the wrong one or oscillates between them.
• A tool description is under one sentence or omits when to use it, what it returns, or the shape of its arguments.

Apply it

1. Write each description like API docs for a new engineer: what it does, when to use it and when not to, expected inputs, and a sample return.
2. Disambiguate overlapping tools by stating in each description what it covers that the others do not.
3. When tool selection is unreliable, rewrite the descriptions before changing the model or adding routing logic.

Sources and further reading

• Writing Tools for Agents · Anthropic
• Tool use (Anthropic Documentation) · Anthropic

The principle

If an instruction has produced the wrong result twice, writing it a third time more carefully rarely helps, because prose is always open to interpretation. Two or three concrete input and output examples kill the ambiguity that no amount of careful description can. Examples show the rule. Prose only describes it.

Why it happens

Examples pin down what prose leaves open. A written instruction can still be interpreted several ways; two or three input-output pairs show the boundary directly. This is especially useful for edge cases, blanks, rejections, and near-misses. The caveat is that examples are powerful but blunt. Order and formatting can over-anchor the model, so test them the way you test code. If you have rewritten the same instruction twice and the failure remains, stop adding prose. Show the behavior you want.

Watch for

• You have rewritten the same instruction two or three times and the output is still wrong in the same way.
• The model handles the typical case but mangles edge cases the prose tried to describe in the abstract.
• Reviewers keep disagreeing about what the instruction actually means, which means the model cannot resolve it either.

Apply it

1. Replace failed prose with two or three labeled input-output examples that demonstrate the exact rule.
2. Include the hard cases explicitly: edge cases, the empty or null case, and a near-miss that should be rejected.
3. Vary or shuffle example order when testing, since order alone can shift results, and keep the examples consistent in format.

Sources and further reading

• Multishot Prompting (Anthropic Docs) · Anthropic
• Language Models are Few-Shot Learners · Brown et al., 2020
• Fantastically Ordered Prompts and Where to Find Them · Lu et al., 2022

The principle

Models are routinely confident and wrong, and unconfident and right. Routing decisions on self-reported confidence inherits that miscalibration. 'Only flag high-confidence issues' or 'be conservative' just moves the noise around. It doesn't reduce it, because the confidence itself is the unreliable signal.

Why it happens

Verbal confidence is not the same as calibrated probability. A model saying it is very sure often reflects style, not measured uncertainty. Post-training can make this worse because helpful, confident answers are rewarded even when the confidence is not earned. Token probabilities or agreement across independent runs may carry useful signal, but the sentence I am 90% sure is weak evidence by itself. Do not route high-stakes decisions on self-rated certainty. Use observable criteria, external checks, or sample agreement instead.

Watch for

• Your gate is phrased as only act on high-confidence outputs or be conservative rather than as concrete criteria.
• Spot-checks turn up confident wrong answers and hesitant right ones at similar rates.
• Two cases that are equally clear-cut to a human get very different self-reported confidence from the model.

Apply it

1. Replace confidence thresholds with explicit categorical rules for what counts as in and what counts as out.
2. Anchor each rule to observable features of the input, with one worked example of an included and an excluded case.
3. If you need a real uncertainty signal, derive it from agreement across independent samples or an external check, not from the model's self-rating.

Sources and further reading

• Language Models (Mostly) Know What They Know · Kadavath et al., 2022
• GPT-4 Technical Report (calibration, Figure 8) · OpenAI, 2023

The principle

Faced with two plausible matches, conflicting sources, or a missing field, an agent's instinct is to pick the most likely option and move on, a confident choice that quietly buries the doubt. When the stakes touch identity, money, or anything you can't undo, a quiet wrong guess is far worse than an honest 'this is unclear'.

Why it happens

Models are biased toward producing an answer. When two matches look plausible or a field is missing, the easy path is to pick one and move on. That hides the uncertainty from every downstream system. Abstention benchmarks show that even strong models often fail to say a question is unanswerable unless the output format allows it. The fix is structural: unclear must be a valid result, not a failure to comply. Give the agent an explicit abstain, unknown, or escalate path and reward it for using that path when evidence is weak.

Watch for

• The agent commits to one of several plausible matches without recording that alternatives existed.
• A required field is always filled, even when the source data plainly lacks the value.
• Conflicting sources get silently reconciled into a single clean answer with no trace of the disagreement.

Apply it

1. Give the agent an explicit way to abstain or escalate, and make unclear a valid, low-friction output.
2. On a tie or a conflict, preserve every candidate with its source instead of collapsing to one.
3. For irreversible or identity, money, or safety-critical decisions, route ambiguity to a human or request a second identifier before acting.

Sources and further reading

• Building Effective Agents · Anthropic
• AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions · Kirichenko et al., 2025

The principle

An aggregate metric is a blended story that smooths over exactly the failures you most need to see. A system at 97% overall can be 99% on the easy cases and 60% on the rare, hard segment where the errors actually cluster. Trust the headline number and you'll automate straight into the cracks it's hiding.

Why it happens

A headline score is a blend. It can look excellent while a small but important segment is failing badly: 99% on common easy cases and 60% on rare hard cases can still average near 96%. Errors are rarely uniform. They cluster by language, intent, customer type, field, document format, or edge condition. Random samples often miss those slices because they are rare by definition. Disaggregated evaluation exists to stop that blindness. Slice the score, oversample the risky cases, and make the worst segment visible before you automate.

Watch for

• You are deciding to ship or automate based on one overall accuracy or pass-rate number.
• Your evaluation set is sampled randomly, so rare high-stakes cases barely appear in it.
• You cannot say how the system performs on your worst segment because you have never measured it separately.

Apply it

1. Break performance down by type, segment, and field, and require every slice to clear the bar, not just the average.
2. Oversample rare and high-stakes cases deliberately instead of relying on a random draw.
3. Treat any slice that falls below threshold as a blocker even when the headline number looks healthy.

Sources and further reading

• Your AI Product Needs Evals · Hamel Husain, 2024
• Designing Disaggregated Evaluations of AI Systems · Barocas et al., 2021

The principle

The common root cause of failed LLM products is the absence of solid evals. Teams ship on vibe checks, iterate blind, and can't tell whether a prompt change improved anything. Manual spot-checking doesn't survive scale or a second engineer. Evals are to AI products what unit tests are to software: the up-front cost that makes every later change cheap and safe.

Why it happens

Vibe checks do not repeat. They tell you whether one person liked a few outputs today, not whether the system improved. Generic similarity metrics rarely capture the product-specific thing you care about, so real progress needs task-specific checks you can rerun. The analogy to unit tests is direct: the up-front cost of an eval harness makes every later prompt, model, or retrieval change safer. Without it, you are iterating on memory and taste. In a non-deterministic system, that usually means trading one unseen failure for another.

Watch for

• Prompt changes are judged by eyeballing a few outputs in a playground and nodding.
• Nobody can state whether last week's change actually helped, only that it felt better.
• A second person tweaks the prompt and silently regresses cases nobody re-checked.

Apply it

1. Stand up a small re-runnable eval set before scaling, and run it on every prompt or model change.
2. Turn every that looks wrong moment into a permanent test case with an expected outcome.
3. Prefer task-specific checks over generic similarity scores, since the latter often fail to track real quality.

Sources and further reading

• Your AI Product Needs Evals · Hamel Husain, 2024
• Task-Specific LLM Evals that Do and Don't Work · Eugene Yan, 2024

The principle

Error analysis, reading your app's actual traces by hand to find where it fails, is the single most valuable thing you can do when building with AI, yet teams skip it for dashboards and vanity metrics that climb while users still struggle. You can't write a good eval for a failure mode you've never seen, and you only see failure modes by reading transcripts.

Why it happens

You cannot evaluate failures you have never looked at. Dashboards show counts, but traces show what actually went wrong. The useful loop is simple: read real runs, write notes without forcing them into categories too early, then cluster those notes into recurring failure modes. Those clusters become your evals. Research calls part of this criteria drift: the act of grading outputs reveals what your criteria should have been. If you choose metrics before reading outputs, the numbers can improve while users still feel the system getting worse.

Watch for

• A helpfulness or quality dashboard is climbing while user complaints or churn are not improving.
• Your eval categories were defined before anyone read a single real transcript.
• Nobody on the team can name the top three concrete ways the system actually fails in production.

Apply it

1. Hand-read a sample of real traces, jotting open notes on each failure before counting anything.
2. Cluster those notes into recurring failure categories and let the clusters define what you measure.
3. Expect your criteria to shift as you read, and revise the eval set instead of freezing it too early.

Sources and further reading

• A Field Guide to Rapidly Improving AI Products · Hamel Husain, 2025
• Who Validates the Validators? (criteria drift) · Shankar et al., 2024

The principle

An LLM judge can match human preferences over 80% of the time, but only after you account for its systematic biases: position bias (favoring the first answer shown), verbosity bias (favoring longer answers regardless of quality), and self-enhancement bias (favoring its own outputs). It's a useful instrument, but an uncalibrated one that grades surface features as readily as substance.

Why it happens

An LLM judge is still a model, and models grade surface features. Studies find position bias, verbosity bias, and self-preference for outputs from the same model family. These are systematic offsets, so averaging more judgments does not remove them. A long answer shown first can win for the wrong reasons. The rubric can drift too as people see more real outputs and realize what quality should mean. Use LLM judges, but calibrate them: swap order, control length, compare to human labels, and never let one biased signal decide alone.

Watch for

• One variant wins your A/B tests and it happens to be the longer answer or the one shown first.
• A model is grading outputs from its own family with no independent cross-check.
• The judge's rubric was written once and never validated against human labels on real outputs.

Apply it

1. Swap answer positions and average both orderings to cancel position bias.
2. Control for length so a padded answer cannot win on bulk, and never let a model be the sole grader of its own family.
3. Validate the judge against a set of human-graded examples and refine the rubric until they agree.

Sources and further reading

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena · Zheng et al., 2023
• Who Validates the Validators? Aligning LLM-Assisted Evaluation with Human Preferences · Shankar et al., 2024

The principle

When a measure becomes a target, it stops being a good measure. Optimize hard against any single metric and the agent learns to game its surface form, padding answers to please a verbosity-biased judge or overfitting a fixed eval set, while the underlying capability stalls or even slips. The number goes up. The thing you cared about doesn't.

Why it happens

An eval is a proxy for what you care about. Optimize against one proxy hard enough and the system learns the cheapest way to raise that score: longer answers for a verbosity-biased judge, format mimicry for a rubric, or memorized quirks in a fixed test set. Reward-hacking research shows this is a deep problem with narrow objectives, not a failure of cleverness. A rotating held-out set helps with memorization, but it does not fix a bad proxy. Use fresh cases, diverse signals, and human reality checks before believing the gain.

Watch for

• Your eval score is climbing steadily while real-user complaints stay flat or rise.
• The same fixed eval set has been the optimization target for many iterations.
• Gains appear as longer, more formatted, or more rubric-matching outputs rather than better substance.

Apply it

1. Keep a rotating, held-out eval the optimization loop never sees, and re-validate gains on it.
2. Treat any metric you actively optimize as compromised and cross-check against fresh data.
3. Watch for surface-form gaming such as padding or format-matching, and penalize it explicitly.

Sources and further reading

• 'Improving Ratings': Audit in the British University System · Marilyn Strathern, 1997
• Defining and Characterizing Reward Hacking · Skalse et al., 2022

The principle

LLM systems are non-deterministic and globally coupled, so a prompt tweak that fixes one case can quietly break three others. Rerunning real production examples against a new prompt is the only way to know you didn't break what already worked. Without a regression suite you're stuck in a whack-a-mole loop, rediscovering the same failures release after release.

Why it happens

LLM behavior can vary across runs, even when settings look deterministic, and one prompt change can shift many unrelated cases because they share the same instruction surface. That means a local fix is not proof of global safety. You need to rerun the old cases and observe what changed. Every production failure you fix should become a permanent regression case. Otherwise the same bug returns in a new prompt, model, or retrieval setup, and the team keeps rediscovering old failures as if they were new.

Watch for

• A bug you fixed last release has reappeared because nobody re-ran the old case.
• A prompt tweak aimed at one case silently broke a different, unrelated case.
• You ship prompt or model changes without re-running the previously passing examples.

Apply it

1. Turn every fixed bug into a permanent regression case with its expected output.
2. Run the full regression suite on every prompt and model change before shipping.
3. Because outputs vary run to run, evaluate over repeated runs rather than trusting a single pass.

Sources and further reading

• What We Learned from a Year of Building with LLMs · Yan, Husain, et al., 2024
• Non-Determinism of Deterministic LLM Settings · Atil et al., 2024

The principle

An agent becomes exploitable the moment it combines three things: access to private data, exposure to untrusted content, and the ability to send data out. Any one poisoned input in that pipeline can steer it into leaking your data, with no code vulnerability required. Guardrail prose isn't enough, because the model can't be the security boundary.

Why it happens

The danger appears when three capabilities meet: private data, untrusted input, and an outbound channel. A malicious document, email, or web page can tell the agent to read a secret and leak it through a tool call, URL, image fetch, or message. No memory-corruption bug is required. The model only has to follow the wrong instruction once. Filtering the payload is weak because attackers adapt. Breaking the chain is stronger: remove one capability, isolate the data, or make outbound actions narrow, reviewed, and allowlisted.

Watch for

• One agent context has access to secrets or private records AND processes text from emails, web pages, or user uploads.
• The same agent that reads untrusted input can also send email, make outbound HTTP calls, or write to a shared external store.
• Your only defense against malicious instructions is a system-prompt line telling the model to ignore them.

Apply it

1. For each workflow, enumerate all three capabilities (private data, untrusted input, outbound channel) and confirm whether one agent holds all three at once.
2. If all three are present, break the chain: drop one tool, split the data access from the untrusted-input path, or route the outbound action through human review.
3. Make any externally-communicating action draft-only or allowlisted to known-safe destinations rather than free-form.

Sources and further reading

• The lethal trifecta for AI agents · Simon Willison, 2025
• OWASP Top 10 for LLM Applications: LLM01 Prompt Injection · OWASP Gen AI Security Project
• Design Patterns for Securing LLM Agents against Prompt Injections · Beurer-Kellner et al., 2025

The principle

Prompt injection is an architectural risk, not a typo you patch once. Models don't reliably tell trusted intent apart from untrusted content, and prose guardrails fall apart under pressure. Newer instruction-hierarchy and isolation patterns help, but the safe assumption is that any untrusted content might be speaking with an attacker's intent.

Why it happens

The model reads trusted instructions and untrusted content inside one reasoning process. It may see labels such as system, user, or document, but those labels are not the same as an external security boundary. Instruction-hierarchy work is improving this, and patterns like CaMeL or Dual-LLM make real progress by separating what reads untrusted content from what holds authority. The old weak defense is to ask the model to ignore malicious text. The stronger defense is to keep untrusted bytes away from privileged action paths and enforce authority in code.

Watch for

• Your security model assumes the model will privilege the system prompt over instructions found in ingested content.
• Untrusted documents, tool results, and operator instructions are concatenated into one context with no isolation boundary.
• A red-team test that hides new instructions inside an input document successfully changes the agent's behavior.

Apply it

1. Treat every byte of ingested content as potentially an instruction from an adversary, and design controls around that assumption.
2. Constrain what actions are reachable after the agent has touched untrusted input, rather than relying on instructions to ignore injections.
3. Move authority out of the model: enforce what the agent may do in deterministic code that the token stream cannot rewrite.

Sources and further reading

• LLM01:2025 Prompt Injection · OWASP Gen AI Security Project
• Understanding prompt injections · OpenAI, 2026
• Designing AI agents to resist prompt injection · OpenAI, 2026
• LLM Prompt Injection Prevention Cheat Sheet · OWASP Cheat Sheet Series
• Defeating Prompt Injections by Design (CaMeL) · Debenedetti et al. (Google DeepMind), 2025

The principle

A confused deputy is a privileged program that a caller tricks into misusing its authority. It isn't malicious, just confused about whose intent it's serving. An LLM agent is the ultimate confused deputy: it holds your credentials and tools, but it'll follow injected instructions and carry out an attacker's intent with your authority. The trap is ambient authority. Authority should travel with the request, not sit waiting inside the agent.

Why it happens

A confused deputy is a program that holds legitimate power and is tricked into using it for the wrong caller. The agent version is direct: it has your tools and credentials, but the intent shaping its next action may come from an attacker-controlled input. The root problem is ambient authority, power sitting in the agent whether the current request deserves it or not. Bind authority to the specific request, caller, and action. Read-only by default and narrow grants for destructive work reduce what an injected instruction can borrow.

Watch for

• The agent runs with a broad, long-lived credential (admin token, write-all API key) it can apply to any action.
• Authorization is checked once at the agent's identity, not per-request against the actual caller and task.
• A tool can perform destructive operations without re-validating that this specific request was authorized for them.

Apply it

1. Default every tool to read-only and grant write or destructive scope only for the specific task that needs it.
2. Bind authority to the request and caller rather than letting it sit latent in the agent's standing identity.
3. Require a fresh, narrowly-scoped grant for any irreversible action instead of reusing an ambient credential.

Sources and further reading

• The Confused Deputy · Norm Hardy, 1988
• An Introduction to Google's Approach to AI Agent Security · Google, 2025

The principle

The Dual-LLM pattern splits the agent in two. A privileged model holds the tools and plans actions but never sees untrusted content. A quarantined model processes the tainted data but has no tools and returns only opaque variables. The privileged model directs the quarantined one without ever ingesting the bytes that could carry an injection. The separation is what makes it safe.

Why it happens

Quarantine works by topology, not by detection. A low-privilege component reads the untrusted page, email, or document and returns structured values. The privileged planner never sees the raw prose; it works with references, ids, labels, or typed fields. That means the attack text has no direct path into the context that chooses tools. CaMeL takes this further with capability tracking in a constrained interpreter. The pattern is powerful but heavier to build, so reserve it for workflows where untrusted content and privileged actions would otherwise meet.

Watch for

• The same model instance both reads scraped or user-supplied content and decides which privileged tools to call.
• Raw untrusted text flows directly into the context that holds tool access.
• There is no structured boundary forcing untrusted content to become opaque variables before the planner sees it.

Apply it

1. Separate the component that reads untrusted content from the component that can take privileged actions.
2. Have the reader return only structured, opaque results (ids, labels, typed fields), never raw text the planner ingests.
3. Let the privileged planner orchestrate by reference, so an injection in the source has no foothold in the acting context.

Sources and further reading

• Design Patterns for Securing LLM Agents against Prompt Injection · Simon Willison, 2025
• Defeating Prompt Injections by Design (CaMeL) · Debenedetti et al. (Google DeepMind), 2025
• Design Patterns for Securing LLM Agents against Prompt Injections · Beurer-Kellner et al., 2025

The principle

Defense in depth means planning for the injection that succeeds. Box the agent in with filesystem isolation (access scoped to specific directories) and network isolation (exfiltration blocked), and a compromised agent can't reach past its sandbox. Real incidents, like CI agents that could leak secrets through untrusted content, show why that second layer matters when the first one fails.

Why it happens

Assume one prevention layer fails. A sandbox limits the damage when it does. Filesystem isolation keeps the agent inside the task directory instead of the whole machine. Network isolation prevents a compromised run from posting secrets to arbitrary hosts. Credential isolation keeps ambient tokens out of reach. These controls are boring and deterministic, which is exactly why they matter. Prompt-injection defenses may reduce the chance of compromise; sandboxing reduces the blast radius after compromise. The second layer is what turns a bad instruction into a contained incident.

Watch for

• Agent tool execution runs with the full host environment, including credentials in environment variables.
• The agent has unrestricted outbound network access rather than an allowlist of required destinations.
• A successful injection could read or write files well outside the task's intended working directory.

Apply it

1. Run tool execution in an isolated environment scoped to a single working directory with no access to ambient secrets.
2. Enforce an egress allowlist that blocks all outbound traffic except the specific destinations the task requires.
3. Design assuming the injection succeeds, and verify that the worst reachable outcome is contained, not catastrophic.

Sources and further reading

• OWASP Top 10 for LLM Applications · OWASP Gen AI Security Project, 2025
• An Introduction to Google's Approach to AI Agent Security · Google, 2025

The principle

The simplest solution that works is usually the right one, and sometimes that means not building an agentic system at all. Agents that direct their own tool use trade latency, cost, and predictability for autonomy, while a workflow with predefined code paths is cheaper and more reliable for well-defined tasks. Reach for an agent only when the problem genuinely needs the model making decisions at runtime.

Why it happens

An agent loop adds cost each turn: latency, tokens, state drift, and another chance to choose the wrong branch. If the task has known categories and a known decision structure, put that structure in code. Use the model for the ambiguous judgment inside the workflow, not for rediscovering the workflow every run. A five-way routing task is usually a classifier plus a switch, not an autonomous planner. Reach for open-ended agents when the branches cannot be listed ahead of time and runtime judgment is genuinely needed.

Watch for

• You can enumerate the possible paths in advance, yet the agent rediscovers them with model calls each run.
• The agent sometimes produces an action or category that does not exist in your fixed set of options.
• Per-item latency and cost are dominated by reasoning steps that always reach the same small set of outcomes.

Apply it

1. Default to a deterministic workflow with explicit code paths for any task whose branches you can list ahead of time.
2. Use the model only for the ambiguous judgment inside the workflow, not for control flow you could script.
3. Promote to an agentic loop only after you confirm the branching is genuinely open-ended and cannot be enumerated.

Sources and further reading

• Building Effective Agents · Anthropic
• Don't Build Multi-Agents · Walden Yan (Cognition), 2025

The principle

Most queries don't need your most powerful model. Routing requests through a cascade, a cheap model first and a stronger one only when confidence is low, can match top-tier quality at a fraction of the cost. The price gap between models spans two orders of magnitude, so paying top dollar for every call is pure waste.

Why it happens

Most requests are easier than your hardest benchmark. A cascade exploits that by trying a cheaper model first and escalating only when a router or validator says the answer is not good enough. FrugalGPT showed large benchmark savings, and later routing work shows the same basic economics. The hard part is not calling the cheap model. It is knowing when to trust it. Self-reported confidence is weak, so validate the router against your own evals. A cascade saves money only if it escalates the cases that truly need help.

Watch for

• Every request hits your most powerful model, including high-volume classification or lookup tasks a small model handles.
• You have no measured deferral signal deciding when a cheap answer is good enough to keep.
• Cost scales linearly with traffic and the easy majority of queries dominates the bill.

Apply it

1. Answer first with the cheapest model that clears your eval bar, and escalate only on failed or low-signal cases.
2. Build a deferral check (a validator or learned router) rather than trusting the model's self-reported confidence.
3. Validate the cascade against a labeled eval set to confirm escalated cases are the ones that actually needed the strong model.

Sources and further reading

• FrugalGPT: Using LLMs While Reducing Cost and Improving Performance · Chen, Zaharia, Zou, 2023
• RouteLLM: Learning to Route LLMs with Preference Data · Ong et al., 2024

The principle

A multi-agent research system can burn roughly 15 times the tokens of a single chat, and token usage alone can explain most of the difference in performance. So multi-agent only makes economic sense when the task is high value and the work genuinely parallelizes. For most tightly coupled work, the coordination overhead isn't worth it.

Why it happens

Multi-agent systems can work, but they are expensive in tokens, latency, and coordination. They pay off when the task is high-value and naturally parallel: independent research threads, separate files, separable hypotheses. They struggle when the work is tightly coupled and sequential, because agents wait on each other while duplicating context. There is also a reliability cost: each handoff can drop constraints or split state. Use multiple agents when the work fans out cleanly. For a narrow sequential task, one well-scoped agent is often cheaper and safer.

Watch for

• The work is sequential or tightly coupled, so sub-agents mostly wait on each other rather than running in parallel.
• Token cost has jumped severalfold after splitting into multiple agents with no measurable quality improvement.
• Sub-agents make conflicting decisions because each sees only a fragment of the shared context.

Apply it

1. Reserve multi-agent architectures for high-value tasks that genuinely parallelize into independent threads.
2. For tightly-coupled work, keep it a single well-prompted agent rather than paying the coordination tax.
3. If you do split, share full traces and constraints across sub-agents so they do not make conflicting decisions.

Sources and further reading

• How we built our multi-agent research system · Anthropic, 2025
• Don't Build Multi-Agents · Walden Yan (Cognition), 2025

The principle

Any system's structure ends up mirroring the communication structure of the organization that built it. For AI, that means if three teams each own a model, you'll get three agents and a brittle seam between them, whether or not the problem wanted to be split that way. The agent boundaries you ship will trace your team boundaries unless you fight it on purpose.

Why it happens

Systems tend to mirror the teams that build them. If three teams each own a model, the product often becomes three agents with handoffs between them, even when the user problem wanted one coherent flow. Those handoffs then become the places where state, ownership, and accountability break. The Inverse Conway Maneuver is the practical answer: shape ownership to match the architecture you want. For agents, that means deciding whether a boundary reflects the work itself or just the org chart that happened to fund it.

Watch for

• Agent or service boundaries line up exactly with team ownership rather than with natural seams in the problem.
• Most production bugs cluster at the handoffs between components owned by different teams.
• A task that wanted to be one coherent flow was split because no single team owned the whole thing.

Apply it

1. Before committing boundaries, check whether each one reflects the problem's structure or just your reporting lines.
2. Where a boundary serves the org chart but not the problem, reshape team ownership to match the architecture you want.
3. Treat the seams between components as the highest-risk surface and design explicit contracts there.

Sources and further reading

• How Do Committees Invent? (Conway's Law) · Melvin Conway, 1968
• Conway's Law · Martin Fowler, 2022

The principle

Agents retry on timeouts, rate limits, and transient errors, but a failed call that never returned may have already succeeded on the server. Without an idempotency key, the retry that 'fixes' a network blip quietly double-charges the card, double-sends the email, or double-books the room. Safe retries depend on the server being able to dedupe.

Why it happens

The dangerous retry is the one after an ambiguous failure. The server may have charged the card, sent the email, or created the record, while the client only saw a timeout. Retrying without dedupe repeats the side effect. Idempotency keys solve this by naming the logical action: the server stores the first result and replays it for later attempts with the same key. Backoff and jitter matter too, because synchronized retries can overload the dependency you are trying to recover. Any retryable side effect needs both controls.

Watch for

• A side-effecting tool call is retried on timeout with no key that lets the server recognize a duplicate.
• Retries fire immediately or on a fixed interval rather than with exponential backoff and jitter.
• You have seen duplicate charges, emails, or records traced to a network blip rather than a logic bug.

Apply it

1. Generate a unique idempotency key per logical action and send it on every side-effecting call so the server can dedupe.
2. Never let the agent blindly retry a non-idempotent operation without that key.
3. Retry with exponential backoff and jitter so synchronized retries do not amplify load on a struggling dependency.

Sources and further reading

• Making retries safe with idempotent APIs · AWS Builders' Library
• Idempotent requests (Stripe API Documentation) · Stripe

The principle

A downstream model or tool that's timing out doesn't get healthier by being called more. It gets worse, while your agents pile up holding open connections and burning their latency budget. A circuit breaker wraps the call so that once failures cross a threshold it trips, and further calls fail fast instead of hanging, which gives the dependency room to recover.

Why it happens

A failing dependency does not heal because agents keep calling it. More calls add load, tie up connections, and push the failure outward. A circuit breaker counts failures and opens after a threshold, so later calls fail fast instead of hanging. After a cooldown, a small probe checks whether the dependency has recovered. The point is not elegance; it is containment. A predictable fast failure can be handled. A thousand slow hangs become an outage. Pair the breaker with retry shedding so recovery is not drowned by traffic.

Watch for

• When a downstream model or tool slows down, your agents respond by retrying harder and connections pile up.
• A single failing dependency drags whole-run latency toward your timeout ceiling instead of failing fast.
• There is no fast-fail path: calls to a known-sick dependency hang until they time out individually.

Apply it

1. Wrap every external model and tool dependency in a breaker that opens after a failure threshold and fails fast.
2. After a cooldown, let a single probe test recovery before resuming full traffic.
3. Shed retries and traffic upstream when load exceeds capacity so retries do not amplify the cascade.

Sources and further reading

• CircuitBreaker · Martin Fowler (after Nygard, 'Release It!')
• Site Reliability Engineering, Ch. 22: Addressing Cascading Failures · Google SRE

The principle

Automation doesn't shrink the human role. It reshapes it into the hardest parts: passive monitoring plus rare, high-stakes intervention. Worse, by taking over the routine work, automation erodes the very skills and situational feel the operator needs when control finally lands back in their lap. You design away the easy 95% and leave humans the 5% they're now least ready to handle.

Why it happens

Automation often removes the easy work and leaves the human the cases the system could not handle. Those are the hardest cases by definition. Over time, the human also gets less practice on the routine work, so their skill and situation awareness decay. When the system finally hands back control, the person is asked to solve a rare, messy problem with cold context. The leftover human job is not smaller. It is harder. Design the handback, keep skills warm, and pass enough state for a real takeover.

Watch for

• The human in the loop only ever sees the cases the agent already failed on, with no exposure to normal runs.
• When the agent escalates, it hands over a half-finished result with no explanation of what it tried or why it stopped.
• The people meant to supervise the agent can no longer do the task manually because the agent has done it for months.

Apply it

1. Route a sample of ordinary, successful cases to the human too, not just the exceptions, so their skill and context stay warm.
2. On every handback, attach the full reasoning trace and a plain statement of exactly what is stuck and why.
3. Design the escalation moment deliberately: make it rare, unambiguous, and accompanied by enough context to act on.

Sources and further reading

• Ironies of Automation · Lisanne Bainbridge, 1983
• From Here to Autonomy: Lessons Learned From Human-Automation Research · Endsley, 2017

The principle

Give people an automated aid and they make errors of omission (missing problems it didn't flag) and commission (following its recommendation even when their own valid evidence says otherwise). The automation becomes a shortcut that replaces careful checking, so the agent's recommendation doesn't just inform the human. It overrides their independent judgment.

Why it happens

Automation bias is what happens when the machine's verdict replaces independent checking. People miss problems the system did not flag, and they follow recommendations even when other evidence disagrees. The driver is cognitive economy: verifying takes effort; accepting the recommendation is easy. High reliability can make the bias stronger because each correct call trains people to stop looking. The interface matters. If it shows only the conclusion, it invites rubber-stamping. Show the evidence beside the verdict and make disagreement as easy as agreement.

Watch for

• Reviewers approve the agent's recommendation at near-100% rates, far faster than it would take to actually inspect the evidence.
• The interface shows the verdict prominently but buries or omits the raw signals the verdict was based on.
• Disagreeing with the agent takes more clicks or justification than agreeing with it.

Apply it

1. Present the raw evidence next to the recommendation, never the verdict alone.
2. Make disagreement a frictionless, one-step action that needs no special justification.
3. Periodically withhold the recommendation entirely so the human has to form an independent judgment.

Sources and further reading

• Does automation bias decision-making? · Skitka, Mosier, Burdick, 1999
• Automation Bias in Intelligent Time Critical Decision Support Systems · Cummings, 2004

The principle

Autonomy is a spectrum, from 'the computer suggests' to 'the computer acts and then tells you' to 'the computer acts and decides whether to tell you at all'. The highest levels are a bad idea for consequential actions, because no aid is perfectly reliable and the cost of a confident error has no ceiling. Autonomy isn't one switch. It's a dial you set per action, based on how reversible and costly that action is.

Why it happens

Autonomy is not one switch. A system can gather information, analyze it, recommend an action, act after approval, or act alone. The right level depends on reliability, reversibility, and cost of error. A receipt resend and a large refund should not share the same autonomy setting. Too much autonomy causes costly silent mistakes; too little creates approval fatigue and rubber-stamping. Set the level per action. Let cheap reversible actions run, and require confirmation where the blast radius or irreversibility justifies human attention.

Watch for

• The agent uses one autonomy setting for everything, so resending a receipt and issuing a large refund run through the same path.
• Irreversible or high-cost actions execute before any human can see them.
• Humans are buried in approval prompts for trivial, reversible actions, training them to click through blindly.

Apply it

1. Classify each action by reversibility and blast radius before deciding its autonomy level.
2. Let cheap, reversible actions run fully autonomously and gate costly or irreversible ones to propose-and-confirm.
3. Tune the dial per action rather than flipping one global approval flag for the whole agent.

Sources and further reading

• Human and Computer Control of Undersea Teleoperators · Sheridan & Verplank, 1978
• A Model for Types and Levels of Human Interaction with Automation · Parasuraman, Sheridan & Wickens, 2000

The principle

Flexible, multi-mode automation produces 'automation surprises', where the system does something unexpected because the operator lost track of which mode it was in, what it would do next, and why. As autonomy grows, the human's job shifts to tracking that state, and every hidden mode change becomes a latent failure path. An agent that silently changes how it behaves leaves its supervisor one step from being wrong about it.

Why it happens

Automation surprises often start when the human loses track of the current mode. The system is planning, then acting; drafting, then sending; reading, then writing. If the transition is silent, the supervisor reasons about the wrong system. Human-factors work found this pattern in high-stakes automation long before LLM agents. Agents add new modes through tool policies, autonomy levels, and hidden state. Make the current mode and next intended action visible. A mode change that changes risk should be impossible to miss.

Watch for

• The agent changes behavior, such as switching from drafting to executing, without surfacing that the switch happened.
• A supervisor cannot answer what mode is it in and what will it do next from the current display.
• Post-incident reviews repeatedly conclude I thought it was still just proposing.

Apply it

1. Keep the current mode, active constraints, and next intended action continuously visible.
2. Make every mode transition an explicit, loud event the supervisor must see, never a silent switch.
3. Treat any uncommanded change in behavior as a defect to surface, not an optimization to hide.

Sources and further reading

• How in the World Did We Ever Get into That Mode? · Sarter & Woods, 1995
• Team Play with a Powerful and Independent Agent: Automation Surprises on the A-320 · Sarter & Woods, 1997

The principle

Each agent can be flawless on its own and the system still breaks, because the bug lives between them: what got passed, what got dropped, who owned the state. Sub-agents don't inherit context automatically. Anything you don't explicitly hand over simply doesn't exist on the other side.

Why it happens

In multi-agent systems, each agent can behave correctly and the whole system can still fail. The bug lives in the handoff: a constraint was not passed, a source was dropped, state ownership was unclear, or the receiver never validated what arrived. Studies of multi-agent traces find many failures in exactly these coordination gaps. A sub-agent cannot use context it never received. If EU market only is not serialized into the task, it does not exist on the far side. Treat handoffs as contracts, not vibes.

Watch for

• A downstream agent produces output that violates a constraint the upstream agent clearly knew about.
• Nobody can say which agent owns a given piece of state, so it gets dropped or duplicated.
• What crosses a boundary is assumed correct and never validated on the receiving side.

Apply it

1. Define an explicit contract at every boundary listing exactly what must be passed.
2. Hand over the full constraint set and source set deliberately rather than assuming context is inherited.
3. Validate incoming data on the receiving side instead of trusting it survived the trip.

Sources and further reading

• How we built our multi-agent research system · Anthropic, 2025
• Why Do Multi-Agent LLM Systems Fail? · Cemri et al., 2025

The principle

People give an agent freedom the way they give it to a new hire: a little at a time, on reversible things first, loosening the leash only as it proves itself. Both failure modes are real. Over-trust leads to misuse, under-trust leads to a good capability being abandoned. Reliance follows the reliability a system appears to have, not just the reliability it actually has.

Why it happens

Good trust is calibrated trust: people rely on the agent where it is reliable and hold back where it is weak. Over-trust causes misuse; under-trust causes disuse. Both waste capability. Algorithm-aversion research shows people can abandon a useful system after seeing one error, even when they would forgive the same error from a human. The answer is not hype or concealment. Show the track record, expose uncertainty, and widen autonomy gradually. Trust should follow demonstrated competence, not marketing, novelty, or fear.

Watch for

• The agent is given broad write access to high-stakes systems before it has a track record on reversible ones.
• Every single action is funneled through manual approval, and the team is quietly abandoning the tool from fatigue.
• The agent presents strong and shaky outputs with identical confidence, giving users no basis to calibrate.

Apply it

1. Start the agent on low-stakes, reversible actions and widen its blast radius only as reliability is proven.
2. Surface where the agent is reliable versus where it is guessing so users rely on it exactly that far.
3. Avoid both extremes: neither hand it production write access on day one nor gate every trivial action behind approval.

Sources and further reading

• Trust in Automation: Designing for Appropriate Reliance · Lee & See, 2004
• Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err · Dietvorst, Simmons & Massey, 2015

The principle

An agent with no legitimate way to say 'I'm stuck' or 'hand this to a human' will invent a path instead. Cornered with no exit, or forced to fill a required field it has no answer for, it makes up something plausible rather than admit the gap. A confident hallucination is the default when honesty isn't an option.

Why it happens

If the workflow requires an answer and the evidence is missing, the model still has to put tokens somewhere. That pressure turns uncertainty into plausible fabrication. Required fields, no unknown value, and no escalation path make the problem worse because the model must satisfy the schema to continue. Give it a clean exit: nullable fields, explicit unknown, stuck, or escalate states. Then a missing answer becomes an actionable gap instead of a confident lie. Honesty has to be a supported output, not a moral request.

Watch for

• Required fields are never empty, even on inputs where the answer genuinely cannot be known.
• The agent has no action that means hand this to a human or I cannot do this.
• Plausible but wrong values appear in exactly the cases where the source data was missing or ambiguous.

Apply it

1. Give the agent a first-class way out: a nullable field, an explicit unknown, or an escalate-to-human action.
2. Make abstaining cheap and explicitly encouraged rather than something the agent must avoid.
3. Treat a confident answer on missing data as a failure mode to detect, not a success.

Sources and further reading

• Reduce hallucinations (Anthropic Docs) · Anthropic
• Survey of Hallucination in Natural Language Generation · Ji et al., 2023

The principle

Without an external signal, a model mostly fails to self-correct its own reasoning, and often makes correct answers worse by second-guessing them. The model that produced a flawed plan is the same one judging it, with the same blind spots. Real correction needs an outside signal: a tool result, a test that runs, a different model. 'Reflect and try again' on the same model with no new information is theater.

Why it happens

A model asked to judge its own work brings the same blind spots that shaped the first answer. Without new information, reflection often samples another version of the same mistake and can even degrade a correct answer. Self-correction research keeps pointing to the same boundary: gains come from outside signals, not introspection alone. Run the test, call the tool, check the database, ask a fresh model, or compare against evidence. Reflect and retry can be useful only when the second pass receives something new to reason over.

Watch for

• Your correction step is just review your work and fix any bugs with no new input introduced.
• The agent confidently rewrites a correct answer into a wrong one after being asked to reflect.
• A corrected output is trusted without any external check ever having run.

Apply it

1. Separate generation from judgment: never let the producing instance be the sole grader.
2. Feed an external signal into the correction loop, such as a test that runs, a tool result, or compiler output.
3. When using a model to judge, give it a fresh instance with no memory of the original reasoning.

Sources and further reading

• Large Language Models Cannot Self-Correct Reasoning Yet · Huang et al., 2023
• On the Self-Verification Limitations of LLMs on Reasoning and Planning Tasks · Stechly, Valmeekam & Kambhampati, 2024

The principle

When findings get summarized and re-summarized, the claim survives but its source, its date, and its uncertainty quietly drop away, until you're holding an assertion you can't verify or defend. Two sources disagreeing isn't noise to flatten. It's signal to keep. A fact without its provenance is just a rumor that carries itself well.

Why it happens

Summaries preserve claims more easily than they preserve trust. Source, date, uncertainty, and disagreement are the first things to disappear. After a few hops, a hedged finding can become a flat assertion that looks as confident as a verified fact. Grounded-generation work treats attribution as measurable for this reason: claims should trace back to supporting sources. Carry the whole tuple - claim, source, date, confidence - through every step. When sources conflict, keep both sides attributed. The disagreement is reliability signal, not noise to smooth away.

Watch for

• A final report states a figure or claim with no source, date, or confidence attached.
• Two sources that disagreed upstream have silently become a single confident number downstream.
• You cannot trace a claim in the output back to the specific document it came from.

Apply it

1. Carry claim, source, date, and confidence together through every summarization and transformation step.
2. When sources conflict, keep both values with their attributions instead of silently picking a winner.
3. Require that every claim in the final output be traceable back to a specific supporting source.

Sources and further reading

• Effective Context Engineering for AI Agents · Anthropic
• Enabling Large Language Models to Generate Text with Citations (ALCE) · Gao et al., 2023