How to measure an auditor. Ours included.
Anyone can say an audit found holes. The questions that matter are how much it misses and how often it raises a false alarm. AMS-1 is a short, open rulebook for answering both, for any auditor of AI graders, reward functions, verifiers or test environments: a tool, a service, an AI model or a person. It applies to us exactly as it applies to anyone else.
An auditor is measured by what it misses and what it wrongly flags, on an answer key fixed before it runs, with every “proved” claim backed by an execution, and with a scorer who can check the work.
Two error rates, one frozen key, and the receipts.
Every number an auditor publishes under AMS-1 carries its scope: what was measured, on which version, how the sample was drawn, the date, and who checked it. A search result is a floor, never a census.
| Requirement | Passes when | Report as |
|---|---|---|
| 1A frozen answer key | Three kinds of items, fixed before the auditor runs: planted holes of named classes; real holes confirmed by execution, including holes first found by someone other than the auditor; and clean controls: correct graders, and correct solutions that differ from the reference, from at least two sources. The key’s hash and commit date are published. | Key hash and date; item counts by kind and class; how each kind was made. |
| 2Blind mixing | Key items, real cases and controls reach the auditor in the same form. Nothing in a prompt, file name or setting marks which is which. | How blindness was kept, in one sentence. |
| 3Misses, per hole class | Recall for each hole class, then overall. Search recall (the auditor starts from the task alone) is kept apart from confirmation recall (the auditor is handed a candidate wrong solution). A class with fewer than 10 items is reported with its interval and marked “small class”; it is never summarized as a per-class result without that mark. | k of n per class, with a Wilson 95% interval; the number of distinct tasks per class. |
| 4False alarms, on clean items | Clean graders flagged and correct solutions flagged, reported separately, on items the auditor did not help build. | k of n, with the Wilson 95% upper bound when none are flagged. |
| 5Repairs, both directions | If the auditor ships fixes: wrong solutions blocked and correct solutions refused, on solutions not used to build the fix; and the harness the fix is graded in keeps test files, runner settings and result parsing out of the solution’s reach (locked or restored before grading). | k of n each way, with intervals; how the harness is locked. |
| 6“Proved” means executed | A finding is proved only when an execution shows the grader pays wrong work (or refuses right work) and a separate check, not the grader under test, fails it. Claims supported by reading code alone are labelled “supported by reading” and reported apart. Proved findings are replayed in fresh environments. | Replay k of n; proved and read-only counts kept apart. |
| 7Rules before results | Protocol, key hash, sample, stop rules and decision rule carry a timestamp older than the first result. A rule overridden later is published with the rule, both results and the reason. | Protocol hash and commit time; every deviation named. |
| 8Models and judges named | Every model is named by company and exact version, with its role. A judge from the same company as a system it scores is flagged. A model verdict that decides whether a finding counts (for example, whether a behavior is in the task’s scope) is reported as a model judgment. Agreement among models from different companies is a robustness check on that judgment, not ground truth: report it as agreement, never as accuracy. Accuracy of such judgments is measured only against a key or a person’s blind check on a stated sample. Model review is never described as human review. | A table of models, companies and roles; agreement counts, with the models that agreed. |
| 9Independent scoring, or labelled self-scoring | Who built the key, who ran the auditor, who computed the score. Tier I: the key is held and the score computed by a party with no commercial stake in the result. Tier S: self-measured, allowed when labelled and shipped with a kit anyone can re-score. Fees never depend on findings or results. | The tier, the parties, the conflicts. |
| 10Re-runnable and correctable | A kit with pinned versions, the items, the key (released after the run), raw outputs and the scoring script; one command recomputes each number. A public corrections policy: a confirmed error is fixed within one business day and logged with the date, old value, new value and reason. | Kit link and hash; corrections link. |
How big a sample? With no false alarm observed, the Wilson 95% upper bound is 11.4% on 30 clean items, 6.0% on 60, 2.5% on 150 and 1.3% on 300. A perfect score on 6 items of one hole class still leaves a lower bound of 61.0%. Report the interval and let the reader judge; do not stop a sample when the number looks good.
Per class: aim for at least 10 items per hole class before reporting that class on its own; below that, mark it “small class”. Clustering: the Wilson interval treats items as independent, and items from the same task, benchmark or generator are not. When items cluster, report the number of distinct tasks (and benchmarks) beside n, give the count by task where several items share one, and say that the interval does not model clustering, or use an interval that does and name it.
Established practice, applied to auditors of AI graders.
We did not invent these ideas. AMS-1 brings to this field what software-analysis tool evaluation and diagnostic-accuracy reporting already ask for.
- Planted flaws beside clean cases, scored on true and false positives: the NIST SAMATE Juliet Test Suite and the OWASP Benchmark project.
- Planted faults beside real ones: Just et al., “Are mutants a valid substitute for real faults in software testing?” (FSE 2014); Defects4J (ISSTA 2014).
- Sensitivity and specificity reported together, each with its interval: STARD 2015 (Bossuyt et al., BMJ).
- Intervals for proportions: Wilson (1927); Brown, Cai and DasGupta (Statistical Science, 2001).
- Rules before results: Nosek et al., “The preregistration revolution” (PNAS, 2018).
- Why judge models must be named: Panickssery, Bowman and Feng, “LLM evaluators recognize and favor their own generations” (2024).
- Rigor checklists for agent benchmarks: Zhu et al., “Establishing best practices for building rigorous agentic benchmarks” (2025).
What AMS-1 does not do. It does not rank auditors: a relative “who found more” score on shared tasks is welcome beside requirements 3 and 4, never instead of them. It gives no single pooled score, because a miss and a false alarm cost different things. It does not certify graders. It applies the same way to scripts, models, services and people.
Version and licence. Version 1.0, draft for comment, October 19, 2026 (wording revised October 21, 2026). It becomes final only after a public comment period of at least 60 days; comments and the changes they cause are published with it. How one auditor (the author) measures against it is published separately and is not part of the standard. Changes are versioned and dated; a change that would move a published number is a new major version. Licence on publication: CC BY 4.0. Other auditors and researchers are welcome to comment or co-sign, with no commercial terms. Errors are fixed under our corrections policy.
Measure any auditor this way. Start with us.
Our figures, their scope and who checked each one are on our error-rates page. Ask us about any number.