Context Is Finite: Rot, Budget, Altitude

From prompt engineering to context engineering

Prompt engineering asks: how do I word my instructions? Context engineering asks a bigger question: of everything that could occupy the model's context window — instructions, documents, tool definitions, conversation history, tool results — which tokens should be there right now?

For agents this framing supersedes the old one, because an agent's context is rebuilt at every step. Each tool result, each file read, each turn of history competes for the same finite window. The job is curation at every inference step, not a one-time writing exercise.

The guiding principle, straight from Anthropic: find the smallest set of high-signal tokens that maximize the likelihood of your desired outcome. Every clause that follows is that sentence unpacked.

Context rot: why more is quietly worse

Models advertise big windows — hundreds of thousands of tokens — so it's tempting to treat context as free. It isn't. As the window fills, the model's ability to recall any specific fact inside it degrades. This is context rot, and two mechanical reasons drive it:

1. Attention is pairwise. Every token attends to every other token — n² relationships. At 1,000 tokens that's a million pairs; at 100,000 tokens, ten billion. Attention is a fixed budget spread across all pairs, so each individual relationship gets thinner as n grows.
2. Training skews short. Models see far more short sequences than very long ones during training, so they have fewer learned parameters for context-wide dependencies. Long-range recall is their least-practiced skill.

The crucial nuance: rot is a gradient, not a cliff. Nothing breaks at some magic token count — reliability just slides. The fact on line 40,000 is usually still found. "Usually" compounds badly across a 50-step agent run: ten retrievals at 95% each means a coin flip that at least one silently fails.

Claude Code's own docs put it operationally: the context window is the single most important resource to manage, because performance degrades as it fills.

Budgeting: spend tokens like money

Treat every category in the window as a line item and ask what it earns:

Two budget rules with measured backing:

• Position matters at scale. For long inputs (20k+ tokens), put the documents at the top of the prompt and the question at the end — this ordering improves response quality by up to 30% on multi-document tasks. The query lands closest to generation with everything it refers back to already read.
• Cut what can't earn. A few well-chosen examples beat an exhaustive edge-case list; stale tool results are the safest first thing to drop.

Altitude: the Goldilocks zone for instructions

"Altitude" is how abstract your instructions are. Both extremes fail:

Too low (micromanagement): hardcoded if-else logic in prose — "if the user asks about refunds and the order is under 30 days and the item is unopened, then..." Brittle: reality produces cases your rules never covered, and the model follows a nearby-but-wrong rule confidently.

Too high (platitudes): "be helpful and accurate." True, useless — it changes no decision.

The right altitude: principles and heuristics specific enough to steer, flexible enough to generalize. "Refunds within 30 days are approved automatically; outside that, gather order details and escalate — never promise an outcome you can't guarantee." The model handles novel cases by reasoning from the principle rather than pattern-matching a rule list.

A real before/after: an agent's system prompt shrank from 4,000 tokens of edge-case rules to 800 tokens of heuristics plus 4 curated examples — and became more reliable, not less. The rules had been feeding rot and mis-matches; the principles generalized.

Structure helps at any altitude: organize the system prompt into clear sections (background, instructions, tool guidance) so the model — and you, six weeks later — can find what governs what.

Tools compete for the same budget

Tool definitions live in the window too, and they have an altitude problem of their own: overlap. If a human engineer can't say which of two tools applies to a task, the agent can't either — it will split its behavior between them unpredictably. Tools should be non-overlapping and self-contained, and any tool the task can't plausibly need is pure rot-feed.

The operating mindset

Before adding anything to context, ask three questions: Is this high-signal (does it change behavior)? Is it needed now (or can it be fetched just in time — next lesson's topic)? Is it at the right altitude (principle, not micromanagement; substance, not platitude)?

Three yeses, it goes in. Anything less, it's rot waiting to happen.