Your context window needs a budget, not a bigger limit

Every time context windows get bigger, I watch a team celebrate by filling them. Then they file a bug that the agent "gets dumber after a while", as if attention were free and position didn't matter. A big window is a big apartment: you can fill it with boxes you never open, and now you live in a warehouse.

I budget context the way finance budgets money: named line items with caps, and someone has to cry before a cap moves.

the line items

• System prompt and tool definitions: fixed, stable, cache-friendly. If this grows past a few thousand tokens, tools are over-described.
• Task state: the current plan and recent decisions. Small, rewritten in place, never appended forever.
• Working set: the files, docs or rows the current step needs. Loaded for the step, evicted after. This is where the budget fights happen.
• History: recent turns verbatim, older turns compacted or dropped.

The eviction rule matters more than the split. Mine is blunt: anything the next step cannot name a use for leaves the window. It can come back through retrieval or memory files if it turns out to matter. It almost never comes back.

why bother when the window is huge

Three reasons, in descending order of how much they cost me before I learned them. Long contexts degrade retrieval quality in the middle of the window, so the stuff you buried is the stuff the model misses. Input tokens bill linearly even when attention doesn't work linearly. And latency grows with every token you ship, so the lazy window is also the slow window.

the audit, because you cannot budget what you cannot see

The single most useful artifact in my agent codebase is a context dump: one flag that writes the exact assembled window to a file, per turn, with a byte count per line item. The first time I ran it I found a 9,000-token tool result from turn two still riding along at turn forty, a stack trace duplicated eleven times by a retry loop, and a "brief summary" that had grown to the length of the thing it summarized. None of this was visible from the outside. From the outside it just looked like the model got worse.

Run the dump on your longest sessions and sort by line item. The distribution is never what anyone guessed. Tool results dominate, system prompts are fatter than remembered, and the actual user request, the thing all of this exists to serve, is routinely under 2 percent of the window. If that ratio does not offend you, read it again after the invoice arrives.

budgets are per step, not per session

The refinement that took me longest: a single cap for "the context" is too blunt, because different steps deserve different windows. My planner step runs lean, a few thousand tokens, because planning degrades fastest with clutter. Execution steps get the working set for exactly one subtask. Review steps get diffs, never full files. Each budget is enforced in code before the call, not requested in the prompt, because asking a model to ignore what you shipped it is paying for tokens twice: once to send them, once to apologize for them.

the objections file

"Just let the model manage its own context." It does, partially, when you give it context-editing tools, and it is decent at clearing its own tool results. It is terrible at evicting things you told it were important, because you told it they were important. The budget is the adult in the room; the model negotiates within it.

"Caching makes big prompts cheap." Caching makes STABLE prompts cheap. The working set churns by definition, so it never caches, and it is exactly the part the lazy window lets balloon. Cache the fixed line items, budget the churning ones; the two disciplines compound instead of competing.

"We tried budgets and quality dropped." Every time I have chased this it was the eviction rule, not the budget. Someone evicted by age instead of by relevance, and the plan from turn three, still very much in force, aged out while a fresh but useless stack trace stayed. Evict by "can the next step name a use", never by timestamp. Age is not relevance, in windows or anywhere else.

The teams whose agents stay sharp for fifty turns are not the ones with the biggest windows. They are the ones who can tell you, per turn, what is in there and why.

Start with the dump flag this week. It is twenty lines of code, it costs nothing in production when the flag is off, and the first file it writes will pay for the whole exercise. Budgets can come later; you cannot argue about line items until someone prints the receipt.