The detector
that failed
its own test.

SEC. 01

Hardware that lies quietly

Not a crash. Not an exception. The computation finishes, the number is wrong, and nothing anywhere reports a problem. Google traced this to manufacturing defects in CPU cores. Meta reported it at fleet scale. Both found rates higher than software engineers assume.

Error-correcting memory does not help, because the corruption happens inside the datapath rather than in storage. If a multiply produces the wrong result, ECC has nothing to correct.

For machine learning workloads the obvious place to look is matrix multiplication, which is where nearly all the arithmetic happens.

SEC. 02

A 1984 check with a 2026 problem

Algorithm-based fault tolerance, from Huang and Abraham in 1984: compute C = A × B, then verify an identity that must hold in exact arithmetic. Take a vector of ones e and compare Ce against A(Be). Two paths to the same quantity. If they disagree by more than rounding explains, something went wrong. The check costs a small fraction of the multiply it protects.

The catch is the phrase "more than rounding explains".

Floating-point arithmetic makes the two paths disagree even when the hardware is perfect, so the detector needs a tolerance. The classical tolerance inherits a condition from the rounding bound it comes from: Ku ≤ 0.01. At bfloat16 with a contraction length of 4096, Ku is about 32.

This is not hypothetical. Prior work testing checksum ABFT against real corruption on genuinely unhealthy production nodes ran at float32 for exactly this reason, reported false positives on healthy hardware at bfloat16, and named the extension to low-precision datatypes as open work.

SEC. 03

Calibrate instead of derive

I did not derive a better bound. I measured the distribution the bound was supposed to describe, and calibrated a threshold from it.

Collect the residual from roughly 80 million clean per-row checks on correct hardware, fit a Generalized Pareto distribution to the tail using peaks-over-threshold, and extrapolate to the quantile corresponding to a per-GEMM false-positive rate of one in a million. That gives a threshold with a stated error rate behind it, in a regime where the theoretical threshold has gone vacuous.

The per-GEMM to per-row conversion assumes the 4096 row tests are independent. Rather than assert that, I measured it: exceedance counts have variance 39.93 against 40.55 for the matching binomial, a ratio of 0.985. Slightly under-dispersed relative to independent trials, which supports the conversion at the fitting threshold and does not establish independence deep in the tail. That last part is carried forward as a stated assumption rather than quietly dropped.

Total compute spend: zero dollars. Everything ran on free-tier Kaggle T4s.

SEC. 04

76% against a bar of 90%

The detector catches 304 of 400 single exponent-bit flips. The bar was 90%, written into the repository README before any code was written, and never moved.

Two points of provenance, because a threshold-dependent number invites the obvious question. The same quantity is 75.25% at one observed clean maximum and 80% at another. Those differ because an observed maximum is a sample statistic that grows with the sample size, not because any rule or fit changed. I report 76% as canonical because the calibrated threshold is the only one with a stated false-positive rate behind it, and it happens to fall between the two sample maxima, so it is not an artifact of either run.

The verdict was also determined before the fit existed. It was recorded in advance that the extrapolated threshold would land at or above the observed clean maximum, and that detection is monotone non-increasing in threshold, so 75.25% already bounded detection from above at any defensible operating point. The bar was unreachable before the number was known.

SEC. 05

Zero out of six hundred

Low-mantissa flips are detected in zero samples in every cell. And more corruption does not help: one flip, two flips, and four flips produce residuals identical to four significant figures.

I checked whether that was an artifact of my injector rather than a property of the detector, on the theory that a low-mantissa flip might be rounded away by the cast back to bfloat16. It is not. Across 50 samples per class, zero injected flips were discarded and every sample showed a strictly positive relative delta. The perturbations are real and the detector does not see them.

This matters more than the 76% does. A perturbation below the accumulation floor is not recoverable by lowering the threshold, because the threshold is bounded below by the false-positive requirement. It is a blind spot, not a setting.

SEC. 06

The mechanism was wrong

The intuition is natural: a fixed single-element error should get harder to spot as the row sum it is hiding in grows. I believed it firmly enough to design an experiment around it, and I registered the prediction in advance along with a falsifier. If detection stayed flat across a 16x range of contraction length, the mechanism was wrong.

The falsifier fired. Sign-flip detection at K from 512 to 8192: 79%, 75%, 73%, 81%, 77%. No trend. Eight points of spread, consistent with sampling noise.

The clean noise floor does fall with K, at a measured exponent of minus 0.4999 over the full range, which is about as clean a square-root law as you get from real hardware. But the threshold is derived from that floor, and the signal scales against the same growing normalizer. Signal and floor move together, and their ratio is what the detector actually tests.

What governs detectability is which bit flipped, not how long the contraction was. Anyone reasoning about ABFT sensitivity from row-sum magnitude should check whether their normalizer cancels the effect they are predicting.

SEC. 07

Four predictions failed. Four findings were retracted.

Three of the four failed predictions were the mechanism the study was built on. Two of the four retractions I caught by predicting them wrong. Two I caught by auditing my own code and data in the week before publishing, which is a less flattering way to find out.

A comparison operator that discarded the loudest possible signal. I had reported that detection was non-monotone in flip count, meaning more corruption was detected less often, and flagged it as an unexplained finding. It appeared in two separate runs, which I took as corroboration. It was a bug. Exponent flips can produce a non-finite residual, which is the checksum failing as loudly as it can, and a bare greater-than comparison evaluates false on a NaN. Ten of fifteen affected cells contained non-finite residuals, up to 14% of samples. Under the correct predicate the effect vanishes entirely and detection is monotone at every K. The corroborating second run was the same bug.

A results document citing a data file that never existed. Three of the paper's claims rested on a JSONL file at a path that had never matched an actual filename, on a host with ephemeral storage, gitignored per file. I found it, verified it reproduced every published number exactly including the minus 0.4999 exponent, committed it, and recorded the provenance. If I had not gone looking, the paper would have shipped with three claims nobody could regenerate.

A prediction falsified at its precondition. I registered a mechanism to explain the mantissa floor: a flip in operand A perturbs both the numerator and the denominator of the residual and may self-cancel, while a flip in B perturbs the numerator only. Then I found the harness injects into the output tensor only, so no operand asymmetry exists anywhere in the data. The predictions are untestable rather than unmeasured. No pass or fail is claimed, and they are reported rather than deleted.

Writing the bar down first constrains what you may conclude from a measurement. It does not guarantee the measurement is right.

SEC. 08

Scope, stated plainly

All corruption here is injected, at the level of the output tensor, after the multiply completes. Operands are not perturbed. Neither are accumulator intermediates, instruction encodings, or control flow. Instruction-level and micro-architectural effects are invisible to this model, and finer-grained injection studies operate below this abstraction.

Also: one GPU model, one workload, one datatype, one shape family. The noise floor is per-configuration by construction and should not be carried anywhere else.

The fleet corruption literature motivates why a cheap detector would be worth having. It does not supply these numbers, and these numbers do not speak to it.

SEC. 09

What broke, and what I would change

Both kill tests failed and I cut the product rather than the bar. KT-1 failed at 76% against 90%. KT-2 failed on overhead, so assay watch is cut as a product: there is no tokens-per-second delta on any GPU that clears the 10% bar at any sampling rate. The bar was not moved to 25%. The detector was not retuned.

The tool refuses to guess. On uncharacterized hardware assay run prints its measurements and returns INCONCLUSIVE, exit code 2. It never emits PASS or FAIL it cannot support. A false positive would tell a company its expensive GPU is broken, which is a worse failure than a missed detection, so the asymmetry is deliberate.

An earlier residual definition destroyed the signal, and I called it structural. The first version summed the checksum vector to a scalar before comparing, discarding degrees of freedom. Under it, sign and mantissa flips were undetectable in all 1800 samples, and I described that as structural invisibility of the ones-vector checksum. It was not structural, it was the reduction. Under the per-row form, sign detection is 155 of 200 on the same workload, shape, harness, and seeds. The voided evaluation is retained in the repository.

One percentile fit could not be evaluated at all. The data collection stored only the top 64 per-row residuals per GEMM. At the 95th percentile that truncates every single GEMM and loses roughly two thirds of exceedances, systematically the smaller ones. At the 99th percentile the effect is negligible, 2 GEMMs of 19500. This was discovered after the method was locked, so it is reported as a data-collection limitation with two of three refits evaluable, not as a change of method.

An open discrepancy I have not resolved. Two checksum reduction backends disagree, one accumulating in float32 and one in float64. Every result here uses a single float64 path, but the discrepancy is unresolved and a threshold calibrated on one backend should not be assumed to transfer to the other.

What I would change: inject at operand level, not just the output tensor. The one unexplained observation in the paper, a shared minimum ratio across every mantissa cell, has a candidate mechanism that my harness structurally cannot test. I registered predictions about it before discovering that, which is its own small lesson about checking what your instrument can reach before predicting what it will find.