Standards
Every rule an audit is held to, published in full.
A finding is only as good as the rules behind it. These are ours: how an audit proves a finding, what a certificate requires, how we measure and correct our own errors, how owners hear first, and what we never do. Each document carries its version and date, so you can hold us to the exact text. Last updated October 19, 2026.
1. The documents
| Document | What it governs | Version and date | Read |
|---|---|---|---|
| MethodAudit method specification | Every step of an audit and the module that runs it: how wrong fixes are searched for, how each one is proved with a separate behavioral check, how findings are re-run in fresh sealed containers, and how replacement tests are verified on both sides. Where the text and the code disagree, the code is what ran, and we ask you to report the difference.SHA-256 fba39770d56db2947680f02576940c252cbf22b1529dc040a9510951edac5648 | Version 0.3.0 Public edition, October 2026. Version 0.2.0 stays online so older links work. | |
| CriteriaPlumbline Verified criteria | The six criteria a release must meet before we certify it, fixed before any certification audit starts. A release that misses them gets no certificate, never a softened one.SHA-256 2bf437ee51c582858f18c3f26b05b99b2cceb5d0c07156499da28b29351a1fb5 | Version 1.0 Effective November 10, 2026. No certificate has been issued yet. | |
| IndependenceIndependence policy for certifications | The rules that keep a certificate apart from repair work: a certificate always rests on a fresh audit, run separately from any audit that wrote or verified the release’s replacement tests.SHA-256 d6fd3cc95c51d146257a130c4a36c28c96afa0887dd26522c65955e9284ad200 | Public edition October 2026 | |
| Error ratesHow we measure ourselves | Our own error rates, measured both ways: correct work we wrongly flag and wrong work we miss. Each figure carries its scope, the date, the run and who checked it: a script, an AI model or a person. The headline rates carry 95% intervals. A list of what we do not yet measure stays on the page until a measurement replaces each line. | Web page Measurements from October 3 to October 18, 2026 | Read |
| CorrectionsCorrections policy and log | Anyone can report an error. We confirm or reject it within 5 business days, fix a confirmed error within one business day, and log the date, the old value, the new value and the reason. Corrections are never deleted. | Web page Last updated October 18, 2026 | Read |
| DisclosureDisclosure policy | Private notice to a public task set’s maintainers first, with the evidence and, wherever we can build one, a tested fix; at least 30 days before anything is published; no prices or offers in a notice; right of reply printed beside our findings. | Web page Last updated October 15, 2026 | Read |
| ConductHow we test without teaching AI to cheat | How the wrong code our search writes is contained, why it never becomes training data, the work we do not take on, how automatic safety stops are respected, and who funds us. | Web page Last updated October 15, 2026 | Read |
| SecuritySecurity and data handling | How the tasks you send are isolated, every company that processes them, and how they are deleted. | Web page Last updated October 19, 2026 | Read |
| ReplayReproduction kit | reproduce.sh and its guide. Re-runs any finding we deliver from scratch with bash and Docker: no Plumbline code, no API key, no model.SHA-256 ed6a438ace4459cde493c66d79c319474cb14d3bcdc6b8f9611fbc91dcbd05c4 |
Public edition October 2026 | ZIP |
Check a download against its fingerprint with sha256sum (Linux) or shasum -a 256 (macOS). The public editions differ from our working documents only where those name audited task sets, vendors or run results still under private notice; those references publish from November 10, 2026, as each notice period ends.
2. The rules behind every published figure
- Proof, not suspicion. A wrong fix the grader pays counts as a finding only when a separate behavioral check passes the reference fix and fails the wrong one, again in fresh containers, and the task as written requires the behavior the fix gets wrong (in code audits, a two-of-three vote of AI models from three companies decides that scope). Every finding ships with a one-command replay.
- Both sides, measured. We count correct work a grader refuses as carefully as wrong work it pays, and we hold our own repairs and our own pipeline to the same two-sided test. End to end, 0 of 150 known-correct fixes became findings (95% upper bound 2.5%; model-written correct rewrites on 75 tasks from three public coding task sets). The full table.
- Rules before results. Our clean-control measurement protocol was fixed in writing before the first model call. When we overrode one of our own stop rules, we published the rule, both results and the reason, on the page that carries the figure.
- Scope beside the number. Findings we publish about a public task set come with their scope: which commit, how many tasks, how they were sampled. A count from a search is a floor, what one search proved, and we say so.
- Who checked, said plainly. Every figure says whether a script, an AI model or a person did the checking. We never describe a model’s review as a person’s.
- Owners hear first. Maintainers of a public task set get private notice, the evidence and a replay at least 30 days before anything is published, and their reply is printed beside our findings.
- Errors fixed in public. A confirmed error is corrected within one business day and logged with the old value, the new value and the reason.
- Paid the same whatever we find. Flat fees, fixed before the work starts. Never per finding, never tied to a result. We never sell environments, never certify one we built, and Index numbers are never for sale.
3. Published next
- November 10, 2026: the Grader Integrity Index. Set-by-set results for the 11 public task sets we audited, with how each sample was drawn and which models were used, after each maintainer’s private notice period ends.
- November 10, 2026: Reward Integrity Criteria v1.0. An open floor anyone can check and claim, with evidence, under a Creative Commons Attribution license (CC BY 4.0). The draft outline is on our reward integrity page.
- From November 10, 2026: run-specific references in the documents above, as each notice period ends.
4. Cite or challenge a rule
To cite the method: Plumbline Grader. Audit method specification, version 0.3.0, public edition. October 2026. https://plumblinegrader.com/files/plumbline-method-v0.3.0.pdf
To challenge a rule or a figure: write to shane@plumblinegrader.com with the document, the line and what you think is wrong. A confirmed error goes in the corrections log.
Hold our audit to the same standard.
The free 20-task audit uses the same deep search as a paid audit and comes with full evidence. No cost, no obligation.