I wrote the conditions for abandoning it before I wrote the code.

01What this is actually watching

An agent decides what a tool can do by reading that tool's description. A human approves a server once, early, after reading the same thing. Only one of those two parties re-reads it every session.

Nothing prevents a server from editing that description afterwards. A change from "reads a file" to "reads a file and posts it upstream" is a capability change, delivered without a version bump, a release note, or any signal to the person who approved it. Plumbline exists to make that change visible and provable after the fact.

The design constraint follows immediately: the record has to be verifiable by someone who does not trust me. Otherwise I am asking people to swap a server they cannot audit for a log they cannot audit.

02Kill tests, written first

Before any crawler existed I wrote down two conditions under which the project is wrong and stops. KT-1: if fewer than 1% of crawled servers show any post-publication change over six weeks, the premise is false. KT-2: if the classifier cannot label at least 80% of observed changes correctly as cosmetic while catching at least one capability-relevant change, it adds nothing over sha256sum and is not worth existing.

Writing these first is a defence against myself. Once you have built a crawler, a merkle log, and a classifier, you become extremely creative about why the disappointing result is actually interesting. A threshold set before the data arrives cannot be renegotiated after it.

The repository status says Phase E, evidence. Do not launch. Do not post. Kill-test evaluation waits on six consecutive weekly crawls with signed checkpoints and a hand-labelled set. It is not finished, and it does not claim to be.

03Deterministic, or it is an opinion

Two decisions carry the credibility of the whole thing. First, canonicalization: whitespace, ordering, and encoding differences must not register as changes, or the log fills with noise and nobody reads it. Second, the classifier is deterministic, not a model. The same diff must always produce the same label, because a probabilistic verdict on "did this tool quietly gain a capability" is not evidence anyone can act on.

The log is append-only and merkle-hashed, and the verifier ships as a command anyone can run against any endpoint with the published public key. I am not asking to be trusted; I am publishing something checkable. If my server started lying, the log would fail its own verification and the tooling to prove it is in the same repository.

04The part most crawlers skip

This project observes infrastructure other people run, and it will eventually produce findings about specific servers. So the repository carries a crawl policy, an ethics statement aligned to the Menlo Report, and a disclosure document covering both directions: how I report something I find about someone else, and how someone reports a vulnerability in me.

The crawler itself runs sandboxed, with a pivot_root helper for the unshare sandbox, because pulling untrusted metadata from arbitrary registries into your own process is exactly how you become the story instead of writing it.

These documents are cheap to write and nearly always skipped. Publishing "here is when we would name a server, and here is how we would tell them first" before collecting a single data point is the difference between research and surveillance.

05What I would change

The evidence phase is slow by construction: six consecutive weekly crawls before any kill test can be evaluated, and the repository has an IDEAS.md full of features explicitly marked do not build during Phase E. That file exists because the temptation to build features instead of waiting for data is enormous, and building them would let me avoid ever running the test that could end the project.

If I were starting again I would have built the hand-labelling instrument for KT-2 at the same time as the crawler. Labelling is the bottleneck on the classifier claim, and it is the piece that cannot be parallelised or automated away without invalidating the thing it is measuring.