temperature zero is not determinism
I pinned an eval to exact-match at temp 0 and it flaked anyway. Where the wobble actually comes from and what to assert instead.
I once spent a full afternoon bisecting a "regression" that turned out to be the same prompt, same model, same temperature zero, producing two different sentences on Tuesday and Thursday. Nothing changed. That is the bug report: nothing changed, output did.
Temperature zero means the sampler takes the top logit every time. It does not mean the logits are the same every time. Floating point addition isn't associative, so the same sum computed in a different order lands on a slightly different number, and GPU kernels pick their reduction order based on things like how many other requests got batched with yours. Your tokens depend on strangers' traffic. When two candidate tokens sit a rounding error apart, the argmax flips, and one flipped token early means a whole different sentence after it.
Mixture-of-experts models stack a second wobble on top: routing is part of the computation, and near-tie routing decisions flip the same way. And that's before the provider quietly rotates hardware or kernels under the same model string, which they do not consider a change, because per the docs they never promised you bit-stable logits in the first place.
So stop writing tests that promise it for them. Exact-match asserts belong on structured output with the schema doing the enforcement (contracts, not string comparison). Everything else gets semantic asserts: did it cite the right doc, did the number match, did the classifier land in the right bucket. And any eval that gates a deploy runs n times, because a 96% pass rate measured once is a coin you flipped once.
Temp 0 makes outputs less random. The gap between "less" and "not" is one rounding error wide, and my Thursday was on the wrong side of it.