The price crash quietly turned your 2024 cost hacks into debt

Quick quiz. When did you last reprice the assumptions behind your model router? If the answer contains the year 2024, this post is an intervention.

The pricing trend data is comical at this point: average frontier output price down roughly 88 percent since March 2023. GPT-4 launched at $60 per million output tokens; its successors sit around $10. Claude Opus cost $75 per million in early 2024; current Sonnet-class models run $15. The index went from 100 to 12, and it did that while the models got better.

Here is the uncomfortable part. Every cost optimization you shipped in 2024 was priced against the old numbers, and most of them were not free. They cost complexity, latency, or quality, and you accepted that cost because the API bill justified it. The bill shrank 8x. The complexity did not.

audit these, in order of likely guilt

The cheap-model cascade. Route to a small model, detect failure, retry on the big one. In 2024 this saved real money. Today the big model often costs less than your 2024 small model did, and the cascade still costs you the failure detector, the retry latency, and every case where "detect failure" quietly meant "ship the bad answer". Run the math again; ours flipped last quarter.

Aggressive context truncation. All that machinery that trims, summarizes and re-ranks to squeeze under a token budget, built when input tokens hurt. Between the price drops and caching that pays you back for repeated prefixes, the budget it protects may no longer exist.

The home-grown batch queue that delays user-facing work to save pennies. The pennies got smaller. The user-facing delay did not.

the actual lesson

None of these were mistakes. They were correct engineering against prices that then fell nearly an order of magnitude, which makes them something more dangerous than mistakes: correct decisions that expired. Nobody re-litigates a decision that was right when made.

So put a date on it. Cost assumptions in this business have a shelf life of about two quarters, and the fix is a recurring half-day: pull current pricing, rerun the three spreadsheets that justified your routing, truncation and batching, and delete whatever no longer pays rent. Deleting a working system because its reason evaporated takes more nerve than building it did. Do it anyway. The graph only goes one direction.