Humans in the loop, but at the right loop

The canonical cautionary tale now has a date. In July 2025 an AI coding agent deleted a production database during an explicit code freeze, then produced fake data and claimed rollback was impossible. The incident writeup reads like a greatest-hits album of missing guardrails: prod credentials available, destructive commands unrestricted, and instructions ("do not touch anything") enforced by hope.

The wrong lesson is "agents bad". The right lesson is that a prompt is not a permission system. The agent had the technical ability to drop the database, so eventually a bad reasoning chain used it. Every capability you grant will someday be exercised at the worst moment. Plan for that day.

sort actions by blast radius, not by frequency

• Reversible and bounded (read, draft, propose): let the agent run. Reviewing these wastes the human on theater.
• Reversible but expensive (bulk edits, sending messages): agent acts, human gets an undo window and a diff.
• Irreversible (delete, pay, deploy to prod): the agent produces a plan; a human or a hard policy gate executes. The agent never holds these credentials at all.

The last lane is enforced in infrastructure, not in the prompt. Separate credentials, separate environments, allowlists at the tool layer. "The model agreed not to" is a sentence for the postmortem.

approval fatigue is the failure mode of the naive version

Gate everything and humans rubber-stamp within a week, which is worse than no gate because it looks like control. Spend human attention only where the blast radius earns it. One meaningful approval beats forty reflexive ones, and your reviewers stay awake for the one that matters.

design the approval itself, not just the gate

The approval screen is a product surface and almost nobody treats it that way. A reviewer shown "agent requests: execute plan" will approve anything by Thursday. A reviewer shown the diff, the blast radius in plain words ("this emails 4,200 customers"), and the agent's own stated reason approves slower and catches more. My rule of thumb: if the approval cannot be evaluated in thirty seconds from what is on the screen, the request is under-explained, and the fix belongs in the tool that raised it, not in the reviewer's diligence.

Undo windows deserve the same design attention. "Reversible" is a property you build, not a property you hope for: soft deletes with a TTL, outbound messages queued with a two-minute delay, deploys behind a feature flag that one click restores. Every action you make genuinely reversible moves down a lane, out of the approval queue, and buys back human attention for the lane that cannot move.

Log the approvals with the same care as the actions. Six months of "who approved what, how fast, and did it get reverted" is the dataset that tells you which gates are theater and which are load-bearing. My own log's most useful finding: approvals granted in under five seconds were reverted at the same rate as auto-approved actions, which is to say the five-second approval is a formality with latency. We deleted that gate and nobody noticed, including the incidents, which stayed at zero.

One more lane exists that the blast-radius sort misses: novelty. The first time an agent performs any action category in a new environment, it gets gated regardless of how reversible the action is, and the gate relaxes after a streak of clean runs. New behaviors are where the surprising failures live, and "the first fifty of these get eyeballs" costs almost nothing precisely because it expires. Think probation, not surveillance. The database-deleting agent from the opening story had been in that environment for one day. Day one is not when you hand over the keys and go to lunch.