It failed
three times.
The third found
the other two.

SEC. 01

The problem in plain language

Modern software is assembled from thousands of pieces written by other people. Somebody publishes a list of known security flaws in those pieces. A scanner compares your product against that list and hands you a few hundred results.

Most of those results do not matter to you. The flawed feature was switched off at build time, or the code was never included, or your product does not use that part at all. But somebody still has to write down why each one does not matter, because regulators and customers now ask.

So the obvious idea: automate the check. Open the file that actually ships, look for the broken function, and answer yes or no.

That is the idea this repository tested. It does not work, and the interesting part is exactly where it stops working.

SEC. 02

Two bars, set before any code existed

LADING was refusal-first. It emitted a clearance only when it could point at specific bytes on disk and re-derive the reasoning offline. Anything short of that produced "under investigation". No scoring, no heuristic, no model in the decision path.

Of the five justifications the VEX standard permits, three require proving a negative about program behaviour. LADING refuses to emit those at all, enforced by a CI check that greps the source for them. The refusal is the positioning, not a limitation: every competitor sells proprietary exploitability probabilities, and this sells the verifiable subset and openly declines the rest.

Two thresholds went into the README at the first commit. Neither moved across three instrument passes.

SEC. 03

Result 1: it almost never got far enough to look

Out of 15,385 attributed findings across 55 scanned directories, 99.60% stopped before any file was opened. Zero findings terminated at a symbol table stage.

The zero in the middle is the finding. Eighty three percent of the binaries in the corpus have had their symbol tables stripped out, and the whole design was built around that constraint. It never came up. The tool died much earlier, on the deeply unglamorous problem of working out that the name a Debian scanner uses and the name an advisory uses refer to the same project.

Termination was not uniform either. On the three RPM-based artifacts, the dominant failure was not a bad mapping but no mapping at all: 19, 318 and 61 refusals with no alias entry in existence. The alias seed was Debian-shaped. Identity resolution is not one problem solved once, it appears to need separate work per packaging ecosystem, and a table built against one distribution family underperforms on another without failing loudly.

Identity, not evidence, is the wall. Until that is solved, the interesting question never gets asked.

SEC. 04

Result 2: the check could only return one answer

The tool cleared 35 findings. Every one of them said: the vulnerable function is not in the file, therefore you are safe.

The function really was absent. It was also absent from every copy of that library ever built anywhere, because it is internal and never exported. A shared library presents two distinct surfaces: the export table lists what other programs may call, and the executable code contains everything the library does, most of which is private. The check read the export table and treated absence there as evidence about the code.

The subsystem is compiled in. There are 597 references to it in executable code. The deallocation function for the exact data type the vulnerability operates on is exported. The single thing absent is the one function the check looked for, and its absence was structurally guaranteed before the check ran.

Hand verification of 100 statements found 20 of 20 clearances false and 80 of 80 refusals correct. KT-2's bar was one. It failed on the first statement examined, and would have failed on any of the other nineteen.

The tool did not catch this. Nothing in its design was capable of catching it. It took a person with readelf and an afternoon of suspicion.

SEC. 05

Result 3: nobody checked whether the flaw applied at all

Two of the four flaws in the manifest do not affect the software versions they were decided on. One of them targets a subsystem that did not exist in that release. So the answer was correct, arrived at by reasoning that had nothing to do with why it was correct.

That reclassifies 7 of the 35 clearances as correct by accident, leaving 28 unsound, and corrects 4 of the 20 hand labels, leaving 16. KT-2 still fails, because the bar was one and not a rate. Both columns are published rather than one replacing the other, and the labelled file records the original label alongside a dated amendment rather than overwriting it.

My own supporting evidence had compounded the error. Having already corrected a wrong library check, I went on to count references to the wrong protocol entirely, on a build that did not contain the component in question, and reported those counts as proof.

SEC. 06

The fix was aimed one layer too shallow

After result 2, the rule was changed to refuse unless the manifest asserts the symbol is verifiably exported. Re-running over the corpus converts all 35 clearances to refusals. The tool now emits zero clearances on this corpus.

That is the correct behaviour and it is also not a product. A clearance tool that clears nothing has removed its own reason to exist. It does not restore KT-1 either, because 99.60% of findings still terminate before any binary is opened.

And it did not fix the defect underneath the one it was aimed at. Result 3 was sitting there the whole time.

SEC. 07

A secondary result with more immediate consequence

While assembling the corpus I hit something unrelated in two widely used open source tools. Scan a container image directly and get 409 unique findings. Generate a bill of materials of the same image and scan that, same tool, same database, same day, and get 137. No error, no warning, and the file is schema valid.

The disagreement runs in both directions: 278 IDs appear only in the direct path and 6 only in the document path, so neither result is a superset of the other and treating the direct scan as ground truth overstates what the data shows. Across 22 public images, 14 are affected.

The second one matters more than it looks. Asked for the same inventory in a different standard format, 151 packages become 1 and zero flaws are reported. A zero is indistinguishable from a genuinely clean product when you are looking at it from outside, and some organisations standardise on exactly that format for their regulatory paperwork.

Both were reported upstream with complete reproductions on public images before the paper was prepared, and neither was withheld pending publication. Where an arithmetic error was later found in my own filed report, I posted a correction on the same thread rather than editing the original, so the record shows the correction.

SEC. 08

Why it is archived rather than iterated

KT-1 is recorded as not evaluable rather than fail, and the distinction is deliberate. A fail would mean the idea was tested and lost. What happened is that the test never ran, because the pipeline stopped upstream of the question. Calling it a fail would claim a measurement that was never made. An earlier reading of the same data did label it fail; that label is formally withdrawn and the withdrawn reading is preserved rather than deleted.

Three consecutive not-evaluable readings were recorded, across three instrument passes each intended to restore evaluability. Under the rule written at the first commit, three is a pattern and the action is stop.

Nothing here is maintained and you should not deploy it. The dataset, the paper and the two upstream bug reports are the output.

SEC. 09

What broke, and what I would change

I measured the wrong library, then the wrong path pattern, then evidence from a second copy of a library at a different version on the same image. Three instances of one defect class, in a project whose entire premise was deterministic evidence. The shared mistake is assuming file layout instead of asking the filesystem, and it appears a third time at the level of the corpus itself, where two traversals disagree about what the corpus even is. Both are published as an audit record rather than reconciled away.

My soundness pass was near-tautological before the corpus produced real verdicts. An early reading had KT-2 passing on 100 statements, but every one of them came from constructed fixtures because zero corpus statements reached a verdict. That result meant "sound on inputs I designed". The ground truth was rebuilt from real pipeline output, and that is the version that failed.

Precision and recall of 1.000 is in the data, and printing it alone would be a lie. Per reason code, the refusal rows score perfectly. The clearances carry no reason code and appear in the same table with recall 0.000 and 20 false negatives. An accurate sentence assembled into a false impression is still a false impression, so the two are always printed together.

The corpus was probably the wrong corpus. LADING reads native symbols, and much of the CVE mass in general-purpose container images is Go and npm packages with no symbol table to interrogate. The firmware stratum, where the binary profile was most favourable at 75.5% stripped against 96.6% in containers, is also where identity resolution performed worst. I would run the packaging-ecosystem histogram before writing another line of the mapping layer.

I do not redistribute the scanned artifacts. The corpus contains proprietary vendor firmware whose licences do not permit republishing. The catalogue records a source URL and a SHA-256 for each instead, so reproduction depends on those sources staying up. The hashes let you verify whatever you retrieve; they cannot guarantee you can retrieve it. That is the honest trade against republishing material I have no right to republish.

What I would change: build the version gate first. It is the cheapest check in the entire pipeline, it would have terminated 7 of the 40 decisions before any symbol rule ran, and not having it cost a third of a result and four wrong hand labels.

Corrections are welcome and will be published alongside the original. That is not a courtesy. Two of the three results here exist because somebody went back and checked something that had already been written up.