Temperature 0
is not
deterministic.

SEC. 01

The ship gate that isn't

You bump a model id in CI. Tests still pass. Customers still hit a new failure mode.

Here is the concrete case I kept coming back to. A support agent used to call lookup_order and then refund. After a quiet model change it skips the lookup and refunds from memory. Every unit test mocks the tools, so none of them notice. An LLM-as-judge score might still read "fine". The behaviour changed and you shipped it.

The obvious move is to write a tool that diffs agent behaviour before and after a change. That is what I built. But a diff is only meaningful against a floor. If two identical runs already differ, a tool that reports "these two runs differ" is telling you nothing you can act on.

So the first experiment was not the interesting one. It was the control. And the control turned out to be the finding.

SEC. 02

The instrument

Vernier wraps the command you already run. It does not ask you to edit your agent.

Recording writes an append-only run: event lines plus content addressed blobs. That record is the evidence. Re-driving replays it under a shim, and the shim is where the one interesting decision lives. In vern verify everything is stubbed, model included, which proves the substrate can reproduce a run at all. In vern diff and vern gate the model goes live and only the tools come from the record.

Alignment turns two trajectories into one edit script of matches, inserts, deletes and substitutions. Costs come from a versioned table, not an embedding, so the same inputs produce the same alignment bytes on Linux, macOS and Windows. Then twelve fixed rules read the aligned pair and emit a severity.

There is no model anywhere on that path, and that is the point. A judge that can change its mind tomorrow can give you a signal, but it cannot be your gate. If classification ever needs a coin flip or a network call, Vernier is broken.

SEC. 03

The most boring experiment I could design

Same model. Same prompt, byte for byte. Temperature 0. Seed pinned. Running locally, on a CPU, one request at a time, with nothing else competing for the machine. Every knob held still.

6.7% of runs still came out different.

That number needs its caveat stated before anything else, not buried at the bottom. This is a 4B quantised model on CPU at batch size 1. That is a far more favourable setting than hosted inference under real load, where batching, kernel selection and routing all move underneath you. So treat 6.7% as a floor, not an estimate. Whatever you are actually running is at least this noisy.

SEC. 04

I went looking for my own bug

A number like 6.7% at zero delta usually means you broke something. So I assumed I had, and went hunting. I took every point where two runs diverged, pulled the exact request body, and replayed it 50 times.

The model gave the same answer 49 times and something different once. Roughly a 2% rare path, and it showed up on almost every prompt I looked at.

That shape matters more than the percentage. This is not a model wobbling randomly across a spread of answers. It is one dominant answer with one quiet alternative sitting behind it, waiting. Which means you can run your evaluation suite twenty times, see it pass twenty times, and still be one sample away from the other arm.

SEC. 05

A fork, not a rephrasing

If the rare arm were a reworded sentence, none of this would matter much. It is not.

Different search. Different documents. A different context window for every step that follows. The agent is not saying the same thing differently, it is deciding something different and then acting on it.

A 2% event at step one is not a 2% event by step eight. Step two is already reading a different world.

That is the arithmetic that turns a rare token into 6.7% of whole trajectories. It is also why measuring divergence at the token level tells you almost nothing useful. You have to measure the trajectory.

SEC. 06

Where the variance actually lives

Here is the part I did not expect, and the part I think is worth more than the headline number.

The divergence was not spread evenly across the six agents in the corpus. All of it came from one: the agent where the model writes free text into tool arguments. The agents that pick from a fixed set of options were completely stable across every run.

So this is not "models are random". It is much more specific than that, and much more actionable. Give the model a blank field and you have variance. Give it an enum and you have a decision you can test. If you are designing an agent you intend to gate in CI, that is a schema choice you can make on day one, for free, before you have any of this tooling.

I would not have found that by measuring one agent harder. I found it by measuring six shapes and noticing that five of them sat still.

SEC. 07

The version I got wrong

My first version of this study reported 100% divergence against a threshold I had pre-registered at 20%. For about an hour that felt like a discovery.

It was a broken test harness pointing at a dead hostname. The candidate agent died before it called a single tool. The classifier then did exactly its job and reported that an empty run differs from a real one, on every single fixture, with total confidence.

Every part of the system worked correctly. The result was still garbage. That is the failure mode worth being afraid of, because nothing errors.

The lesson I actually took from it is smaller and more useful than "test your harness". It is this: a result that clears your pre-registered bar by five times is not a triumph, it is an alarm. I had written the 20% threshold down in advance precisely so I would have something honest to be measured against, and then I nearly let a 100% flatter me past it.

RESULTS-v0-RETRACTED.md is still in the repository, at the top level, marked do-not-cite. It stays there. Deleting it would make the repo look better and the record worse.

SEC. 08

What broke, and what I would change

The harness was wrong before the result was right. Covered above. Cost me a write-up and a retraction, and it is the single most useful thing in the repository now.

One run replayed 50 out of 50 identical and still diverged in the experiment. This is my favourite detail in the whole study. The recording itself had captured the rare arm. My baseline was the 1-in-50. Which means a recorded run is a sample, not the truth, and every tool in this space inherits that problem, mine included. If I rebuilt the study I would record each baseline three times and keep the modal trajectory rather than the first one.

I have not answered the original question. KT-1 and KT-2 both read UNEVALUATED and that is not modesty, it is the actual state. Testing whether a real model upgrade changes agent behaviour needs paid API access I have not spent yet. The harness is built, the corpus exists, the runner is written. It is roughly twenty dollars of tokens and a free afternoon. Until it runs, the headline claim of the tool is unproven and I would rather say so than let a good control experiment stand in for it.

Tools are stubbed on re-drive, so this does not test your tools. Live-diff asks the new model what to do and feeds it recorded tool results. That isolates the model variable cleanly, and it means Vernier is silent on whether your live integrations still work. Two different jobs. I would rather do one of them honestly.

Providers are nondeterministic and I cannot fix that from here. Treat live-diff as evidence, not as a mathematical identity proof. When you need a hard reproduce check, use vern verify with everything stubbed.

Everything is public, including the raw replay data and the retraction, so you can check my arithmetic instead of trusting it. If the twenty dollars is pocket change to you, the repo has everything you need to run KT-1 yourself. I would rather the number existed than that I made it up.