Fourteen checks before any number leaves.
Every number, table, chart, notice, page, post, press line, sales sentence and client report we produce must pass these checks before it leaves the company. They are the habits behind our published error rates and our corrections log, written down in one place so nobody skips one, and published so you can hold us to them.
Proof, scope and both sides, every time.
| Check | Passes when |
|---|---|
| E1Rules before results | The protocol, sample, stop rules and the reading we expect were committed to git before the first measurement: the commit is older than the first grader or model call. Any deviation is named in the results as a deviation. |
| E2Both sides, with controls | We measure what is wrongly paid and what is wrongly refused, with controls in both directions: correct alternatives, empty or do-nothing answers, and planted known answers. |
| E3Proof, not suspicion | A finding counts only when a check that never calls the grader under test shows the work is wrong, and the finding replays in fresh containers (reported as “k of n reproduced”). Where a track has a second, differently built check, it must agree too, and we name the tracks that have one. Code findings: the proof is one behavioral check, written by an AI model and run in fresh containers; it passes the reference fix twice, fails the unfixed code and fails the wrong fix twice. Whether the missing behavior is required is a scope label plus a vote of AI models from three companies, reported as model judgment, not as a second check. We never say a code finding was confirmed by two independent checks until a second, differently built check has been measured and adopted. |
| E4Interval and scope beside every number | “k of n” with a Wilson 95% interval (an upper bound when k is 0), and in the same sentence or row: what was measured, which version or commit, how the sample was drawn, the method version, and “a floor” for any search result. Never stated more broadly than measured; never compared across tracks or method versions. |
| E5Rules hold, or the override is public | A stop rule says what counts against the tool and who decides. If a rule fixed in advance is overridden, the rule, both results and the reason are published where the number appears. Never silently. |
| E6Who checked, said plainly | Each figure says whether a script, an AI model (and from which companies) or a person checked it. Model review is never presented as human review. “Experts” or “reviewed” describe only a stated person, on a stated sample. |
| E7Human review, where claimed, is blind with hidden controls | The reviewer is independent, with conflicts disclosed; works blind against a key we keep; meets hidden known-genuine findings and planted non-findings; and the result is reported as k of n with an interval and the sample’s scope. |
| E8Conflicts and notice periods checked | Open notice periods, prospects and customers, and related affiliations are checked on the publication date. No owner is named while its notice period runs. A customer is disclosed beside its row. Fees are never tied to findings or tiers. |
| E9Owners heard first, and able to answer | Private notice at least 30 days before naming, with the evidence and a replay; no prices, offers or sales copy while the notice period runs; replies printed beside the result; a disputed finding re-checked before publication. Details: our disclosure policy. |
| E10Anyone can re-run it | Pinned versions (commits, dataset revisions, image digests, model IDs without aliases), seeds, manifests with a hash of every drawn item, and one command per finding that needs no Plumbline account or model key. The kit is replayed once from the package on a clean machine. |
| E11Claims-checked | Every number has a row in our claims file (id, value, scope, source file and field), and our claims verifier passes before release, or each failure is resolved first. No placeholder is left. |
| E12Outside methods review for anything ranked or tiered | At least one outside reviewer with no conflict read the method before results were published; their comments and our responses are published with it. |
| E13Words | Lead with what we found, built or fixed; caveats as scope, not apology. Never “best”, “#1”, “safest”, “unhackable” or “the true rate”; never “certified” outside a certificate issued under published criteria; never a claim that we coined “reward integrity”; never an invented or inflated number, customer, partner or endorsement. |
| E14Corrections | Errors are fixed within one business day of being confirmed and logged publicly with the date, the old value, the new value and the reason. Our corrections policy and log. |
Which checks apply
- Tables, reports, Index editions and public pages: all 14.
- Private notices to the owners of what we audited: E2–E4, E6, E8–E10 and E13.
- Client reports, which we never publish: E1–E7, E10, E11 and E13.
- Posts, quotes, press and sales lines: E4, E6, E8 and E13, and every number must already have a claims row that passed E11.
In force today, with two scoped items. E10 applies in full once our replay kits publish, planned with our November 10, 2026 launch; until then, the reproduction kit re-runs any finding we deliver to a client. E12 governs ranked or tiered tables, and we have not published one yet. Our Indexes add four rules of their own, published with the Index.
Version and challenges
Version 1.0, adopted October 18, 2026; check E3 reworded October 21, 2026. Public edition, published October 23, 2026: the same 14 checks, without our internal file references, the list of candidate outside reviewers we have not yet contacted, and our working gap list. Changes are versioned and dated on this page.
To challenge a check or a number we published: write to shane@plumblinegrader.com with the page, the line and what you think is wrong. A confirmed error goes in the corrections log. Every other rule we are held to is on our standards page.
See the standard applied to us.
Our own error rates show these checks at work: the scope, the interval, both sides and who checked each figure.