DOC GK-010 TYPE ENGINEERING LOG SUBJECT PLIMSOLL FIELD DATA N = 0

How many
times did you
run it?

Someone tells you their model scores 0.87 on their internal eval, up from 0.81. You cannot check any of it. Not because they are lying, but because nothing about that claim is checkable. The quietest failure is not a moved threshold, it is nine runs and one publication. PLIMSOLL makes the attempt count a published fact instead of an optional disclosure. Spec: 10.5281/zenodo.22107451, released CC0. Code: plimsoll. Live log: plimsoll.gautamkhosla.com.

NINE RUNS. SLIGHTLY DIFFERENT SEEDS, TEMPERATURES, RETRIEVAL SETTINGS. BAR published the other eight are never mentioned NOTHING WAS FALSIFIED. EVERY NUMBER IS REAL. THE CLAIM IS STILL UNCHECKABLE.
FIG 10.1 · Moving a threshold after the fact is the amateur version, and everybody already watches for it.

SEC. 01

The question almost nobody asks out loud

Did they pick the bar before running, or after seeing the number? Is the dataset the same one they used last quarter? And the one that matters most: how many times did they run it?

Run the eval nine times with slightly different seeds, temperatures or retrieval settings. Publish the run that cleared the bar. Omit the other eight. Nothing was falsified and every number is real.

Git commits do not catch this. Signed results do not catch this. A vendor's own eval platform structurally cannot catch this, because it belongs to the vendor.

SEAL rule + threshold + digests YOUR EVAL your tool, your machine ATTEST bind results to the seal VERIFY anyone results.json NEVER UPLOADED WHAT CROSSES THE WIRE digests, metadata, verdicts never: datasets, models, prompts, outputs
FIG 10.2 · A notary, not a lab. No scorers, no metrics, no judge. The eval stack sits outside the system permanently.

SEC. 02

The mark on the hull

A plumb line is a weighted string that defines true vertical. The Plimsoll line is the mark on a ship's hull, set in advance by an independent authority, showing how deep it may be loaded. Anyone standing on the dock can look at the waterline and see whether the ship is overloaded. The mark is public, it was placed before the cargo went on, and it cannot be quietly moved.

Before you run, you write down your metric, threshold, dataset hash and decision rule. It is canonicalized, hashed, signed and appended to a public append-only log. Then you run your eval with whatever you already use. Afterwards you bind your results to the seal, and the engine applies only the rule you sealed.

The architectural commitment underneath is the part I would defend hardest: PLIMSOLL has no scorers, no metrics, no judge, no assessment framework. It never receives a dataset or a model. It is a notary, and that is permanent rather than a v1 limitation.

The verifier also points anywhere. It works against any conforming log, not just mine. A transparency log with exactly one operator is just a database with good manners, so if I vanish, go rogue, or start rewriting history, it is detectable by anyone holding the CLI.

ONE SEAL, FIVE ATTESTATIONS attempt 1FAIL attempt 2FAIL attempt 3FAIL attempt 4FAIL attempt 5PASS the log assigns this number inside a transaction there is no API that sets it VERIFIED WITH DISCLOSURES · no flag suppresses this
FIG 10.3 · Iterating is normal. Hiding iteration is the problem. A badge showing three attempts is blue, not red.

SEC. 03

The attempt ledger

When you publish an attestation, the log assigns the attempt number inside a database transaction. The client cannot pick it. There is no API that sets it. A seal with five attestations shows five attempts, in order, with all five verdicts, forever.

That distinction between iterating and hiding iteration is deliberate and it is a design constraint, not a slogan. If the tool shames iteration, nobody publishes anything and the log dies empty. So the disclosure is mandatory and the presentation is neutral.

A SEALED DECISION RULE edit with a flagno env variableno config fileno paid tierno a test fails if a code path that mutates a sealed field ever appears SUPERSEDE: new seal, required public reason, whole chain reported on every verification AMENDMENT IS IMPOSSIBLE. DISCLOSURE IS MANDATORY.
FIG 10.4 · Not moralism. The moment an override exists, the artifact is worth nothing to the person it was meant to convince.

SEC. 04

There is no override

A sealed decision rule cannot be edited. Not with a flag, not with an environment variable, not with a config file, not on a paid tier. I went looking for a code path that mutates a sealed field and wrote a test that fails if one ever appears.

If your pre-registration turns out to be wrong, you issue a superseding seal. It references the old one, carries a required public reason, and every verification reports the whole chain.

This is the only reason a sceptical third party would accept the output at all.

plimsoll verify RUNS NINE CHECKS AND REPORTS EACH V1 attest signature V2 seal signature V3 seal in log V4 attest in log V5 seal predates V6 dataset matches V7 same verdict V8 attempt count V9 log consistent your own signature on a timestamp proves nothing. you control your clock. V3 is what makes "before" real: inclusion proof against a signed checkpoint. AN UNPUBLISHED SEAL IS A LOCAL NOTE TO YOURSELF, NOT EVIDENCE. THE CLI SAYS SO LOUDLY.
FIG 10.5 · Nothing in the trust path is probabilistic. RFC 8785 canonicalization, fixed-precision decimals with no float64, RFC 6962 Merkle with domain separation.

SEC. 05

Determinism, all the way down

Comparisons use fixed-precision decimals from the original lexical form, never through binary64. A threshold of 0.82 comparing wrongly against 0.8200000000000001 is exactly the kind of bug that would destroy the tool's credibility, so the type system prevents it rather than a review catching it.

Dataset hashing treats a dataset as an unordered multiset of canonicalized rows, so reordering your JSONL does not raise a false "dataset changed" alarm, while duplicating a row does change the hash. Canonicalization is version-tagged so a future change cannot silently invalidate history.

Same seal plus same results equals the same verdict, on any machine, forever.

PRE-REGISTERED AGAINST ITSELF, IN THE FIRST COMMIT KT-1 25 distinct organizations publish a seal in 8 weeks UNEVALUABLE KT-2 10% of 100 evals show iteration or supersession UNEVALUABLE KT-3 one unprompted request for a private log in 3 months UNEVALUABLE PILOT N = 0. THE ONLY ENTRIES IN THE PUBLIC LOG ARE MY OWN FIXTURES. THEY COUNT TOWARD NOTHING, AND RESULTS.md SAYS SO IN ITS OWN SECTION.
FIG 10.6 · Not PASS and not FAIL. Writing anything else would be the exact behaviour this tool exists to make visible.

SEC. 06

What this has not shown

There is no field data. All three kill tests are unevaluable at N equals zero, and the entries currently in the public log are my own fixtures, authored and verified by the same person, against a five-row synthetic dataset. They count toward nothing.

That sits at the top of the README rather than in a footnote, because a tool whose entire purpose is making unstated iteration visible cannot itself round an empty log up to a result.

KT-2 is not evaluable: neither PASS nor FAIL. Writing anything else would be the exact behaviour this tool exists to make visible.

Three further limits are stated in the technical note rather than discovered by a reader. Digest-to-artifact correspondence cannot be verified by a system that declines to receive the artifact. A single-operator log admits equivocation without a gossip protocol. And no empirical study of the underlying phenomenon has been conducted, by me or as far as I can find by anyone else, so the premise that selective reporting occurs at a meaningful rate in machine learning evaluation is currently an assumption.

DESIGN COMMITMENTS · PULL REQUESTS THAT VIOLATE THESE ARE DECLINED never receive, store or transmit a dataset or a model never define, select or bundle an assessment framework never compute a quality metric, score or evaluation result never be the only party able to verify an attestation EACH ONE REMOVES A FEATURE SOMEBODY WILL EVENTUALLY ASK FOR.
FIG 10.7 · Certificate Transparency was not built by a Certificate Authority. A vendor attesting to its own numbers has the same conflict.

SEC. 07

Where this sits

PLIMSOLL is not a new idea. It is an old idea pointed somewhere new. Medicine solved this in 2005, when trial registration before enrolment became a condition of publication, because selective reporting was distorting what everyone believed to be true. Certificate Transparency solved the log half in 2013: make issuance public and append-only, and misissuance becomes detectable without anyone having to be trusted.

The lineage matters more than the novelty. Claiming this was invented here would be both false and weaker, because the prior art is the argument that the mechanism works.

The specification is released CC0 precisely so that nobody has to ask permission to implement it, and so that if I abandon the project someone else can implement it cleanly while existing logs remain verifiable.

COST OF OPERATING THE PUBLIC LOG Cloudflare Pages$0 Cloudflare Workers$0 GitHub Actions and storage$0 total$0 / month A LOG NOBODY CAN AFFORD TO RUN IS A LOG WITH ONE OPERATOR, WHICH IS THE FAILURE MODE.
FIG 10.8 · The cost table is a design output, not an economy. Running your own has to be realistic or the trust argument collapses.

SEC. 08

What broke, and what I would change

The publish path is asynchronous and the first version pretended it was not. A seal is not queryable until the log appends, which takes about a minute. Returning success on submit and letting a user assume inclusion would be a lie of exactly the kind this tool exists to prevent, so the CLI now prints what actually happened and offers an explicit wait flag and a poll command.

I split three packages for reasons that look redundant and are not. The Merkle arithmetic lives separately from the log because the browser verifier compiles to WASM and cannot link SQLite. A second, deliberately independent log server exists only so the test suite can prove verification works against a log that is not mine. If the tests could only verify my own implementation, the neutrality claim would be untested.

A reachable denial of service in the canonicalizer. Found and fixed during hardening. The canonicalizer is the first thing that touches untrusted input on the submit path, which is the worst possible place for it and the most predictable.

The single-maintainer problem is real and I did not argue it away. Current security guidance tells organizations to avoid depending on single-maintainer projects for core functions, and that guidance is correct. The answer is not to ask for trust: the log is clonable and offline-verifiable, releases are signed with cosign keyless and carry SLSA provenance, the dependency surface is two direct dependencies, the spec is CC0, and the verifier already points at any log.

What I would change: run the empirical study first. The premise is that selective reporting happens often enough in machine learning evaluation to be worth a log, and I built the log before measuring the phenomenon. KT-2 is essentially that measurement deferred until other people show up, which means the project's own evidence depends on adoption it has not earned yet. Building the instrument before the observation is the same ordering mistake I have now made twice.

Read the spec ↗ The public log
DOC GK-010 · PLIMSOLL · Spec DOI 10.5281/zenodo.22107451 Previous log: GK-009 · LADING →