Confident
where people
disagree.

SEC. 01

The claim, and the question it skips

Most public tests of Jev used coin flips, dice or tasks with one correct label. Real inputs often do not have one: two careful people read the same pair of sentences and disagree. If a model's confidence is going to decide what software does on its own, the question that matters is simple. When people genuinely disagree, does the confidence go down?

ChaosNLI answers the hard part of that for me. It has 3,113 natural language inference pairs, each labelled by 100 people. I took the 750 pairs with the lowest annotator entropy and the 750 with the highest, equal counts on purpose, and scored each prediction against the share of the 100 annotators who picked Jev's label rather than against a single majority answer.

SEC. 02

One test, fixed in advance

Before any scored request, the preregistration fixed the metric, the analysis and the verdict rule. The result tracks disagreement if the bias-corrected gap in calibration error between contested and clear pairs is at least 0.09 with p under 0.05. It holds if a gap of 0.09 is rejected and the corrected gap is at most 0.02. Anything else is inconclusive, and the plan committed to publishing whichever way it went, including a null and including a result that supports TypeSafe's claims.

Every amendment to that plan is dated 22 to 24 September and was made on simulated and fixture data, before any paid scored run. The frozen plan's hash was submitted to OpenTimestamps at 06:29:11 UTC on 25 September, and the scored run began at 07:01:59.

I have been saying the plan was anchored in Bitcoin 33 minutes before the first scored call. Checking the proof for this log, that is not quite what it shows.

The 33 minutes is the time from submitting the hash to the timestamp servers. The proof only becomes independent once a Bitcoin block commits to it, and the first of the three blocks in the proof, 968519, was mined at 07:14:16 UTC: 12 minutes after the first scored call. What the proof does establish is that the plan existed within 12 minutes of the first call, hours before the raw responses were sealed for analysis at 13:39 UTC. The preregistration record on Zenodo was also deposited after data collection, and says so in its own description.

SEC. 03

The trap in comparing calibration

Binned calibration error is biased even for a perfectly calibrated model, and comparing it between two groups is biased toward finding a difference. In a simulation of 100 perfectly calibrated models, the naive comparison flagged about 1 in 10 as worse on hard items at a nominal 5% rate. A study that skipped this would have found the effect it was looking for whether or not it existed.

The preregistration requires a parametric correction and publishes the raw number beside the corrected one. With the correction, the false positive rate came back to 0.051. It is a form of consistency resampling (Bröcker and Smith 2007; Vaicenavicius et al. 2019), extended with a Beta-Binomial model of how annotators sample labels. I credited that lineage properly only in version 1.1; SEC. 07 has the details.

SEC. 04

The result

On clear pairs Jev was close to calibrated: 91.5% confident on average, with 84.4% of annotators picking its answer. On contested pairs it was 80.7% confident while 46.8% of annotators agreed. Calibration error rose from 0.076 to 0.339.

That clears the preregistered bar of 0.09 by a wide margin. The error is overconfidence, and it is the kind that matters most if a threshold like "act if p is at least 0.9" decides what happens without a person.

SEC. 05

Same question, different answer

Jev can be asked the same thing two ways: Choice, one multiple choice question, or Noul, three yes or no questions. Choice shows the gap clearly. Normalised Noul probabilities from the same calls land at 0.076, between the thresholds, so the preregistered verdict for that arm is inconclusive. TypeSafe's own documentation says the two formats need not agree, and here they do not. Reporting only the arm that cleared the bar would have been easy and wrong.

The other half of the story is less damning. Use Jev's confidence to guess which pairs are contested and it works reasonably well, with an AUROC of 0.744. Confidence does go down where people split. It just does not go down nearly enough.

SEC. 06

One knob cannot fix both

The obvious repair is temperature scaling: one number that rescales every probability. The best single setting, T = 11.94, takes contested pairs from 0.339 to 0.044 and makes clear pairs worse, at 0.198. A deployment cannot fix contested inputs this way without breaking the clear ones it was already getting right.

It is also not a Jev-specific defect. A supervised BART-MNLI model, trained on exactly this task, shows the same pattern with larger errors in both groups. The practical advice that follows is plain: treat the confidence as a ranking of how contested an input is, and set thresholds on your own labelled data.

SEC. 07

What was wrong, and what I would say louder

A bug in my BART baseline client. I found it, fixed it and re-ran the baseline. A secondary analysis also had a search limit that was corrected after data collection. Both are in the paper's list of deviations rather than quietly patched.

Missing related work. Version 1.1, three days after the first release, adds five papers I should have cited, including Baan et al. 2024 on human label variation, and credits consistency resampling as the basis of the bias correction. No data, numbers or conclusions changed.

Two CI checks are still red. The citation checker needs the full git history and misreads one file hash in the preregistration changelog as a commit id, and the workflow linter reports findings in the CI configuration. Neither touches the analysis, the unit tests pass, and the README explains both near the top.

What I would say louder: the timing claim in SEC. 02. Submitted 33 minutes before is true, anchored 33 minutes before is not, and the version on my own site said the second one. One task, one dataset and one model version, jev-1.13.0, is also all this study covers. Newer versions may behave differently, and it measures agreement with people, not some deeper truth. TypeSafe provided $5 of API credit, the run cost about $0.41, and I sent them the paper before publishing.