What AI code review catches, misses, and hallucinates

We put model review on every pull request six months ago. Nobody would turn it off now, and also nobody would let it merge anything alone. Both facts are the point.

what it reliably catches

The mechanical layer, and it is genuinely good at it: the null path you didn't test, the error swallowed by a bare except, the off-by-one in pagination, the copy-pasted function that diverged in one line, the timezone bug lurking in a naive datetime. It reads every line with the same attention, which no tired human reviewer has ever done at 4 p.m. on a Friday. As a lint layer with taste, it earns its API bill.

what it structurally cannot catch

The review only sees the diff. It does not know the migration plan this PR quietly contradicts, the incident from March that made this pattern taboo, or that the "unused" flag is read by a service in another repo. The expensive review comments were always about the decision, not the code: should this exist, does it fit where we're going. That context lives outside the diff, so the model reviews the trees and signs off on the forest fire.

the hallucinated finding is the real tax

A junior reviewer says "I don't get this part". The model says "this races under concurrent access" in the same confident register whether it's right or fabricated. Every fake finding costs an engineer twenty minutes of disproving it, and after a few of those, people start skimming the real ones. We cut the noise by demanding line-anchored findings with a failure scenario, and dropping any category that cried wolf twice.

the config that made it liveable

For anyone tuning their own: we run the model review before the human review, never after, so humans arrive to a diff that is already mechanically clean and spend their attention on the decision layer. Findings post as one batched comment, not twelve separate ones, because twelve notifications train people to mute the bot. Severity comes from a fixed rubric the model must cite, not from its mood. And we track one meta-number weekly: the ratio of model findings that led to a code change. Ours sits around 60 percent. When it dips, we prune categories instead of arguing with the robot, because the robot does not attend retros and cannot promise to do better.

Net: keep it, cage it. It's the best lint we've ever had and the most confident wrong reviewer on the team, in the same tool, on the same PR.