An auditor’s error rates, published.
An audit is worth as much as its false alarms and its misses. These are the measurements we have of our own work: what each one covered, when it ran, the run it came from, and whether a script, an AI model or a person did the checking. Method version 0.3.0. Last updated October 15, 2026.
Headline measurements
findings reproduced when re-run from scratch in fresh containers.
finding checks failed a correct refactor of the reference fix.
scope calls upheld by a two-of-three majority of Claude, GPT and Gemini.
correct graders flagged on our agent-environment calibration bench.
No figure on this page has been reviewed by a person yet. Each one says which checks a script did and which an AI model did. The first three come from our October 3, 2026 accuracy study of 9 audits of public coding task sets; the fourth from a synthetic bench we built, with every label frozen before the audit ran.
Are our findings right, and do they hold?
A finding is a wrong fix the task’s own grader pays in full, proved wrong by a separate behavioral check. These four measurements test that chain from four directions.
| Figure | What it measures, and on what | Date and run | Who checked |
|---|---|---|---|
| 424 of 425findings reproduced | Every in-scope finding from 9 audits of public coding task sets, re-run from scratch in fresh containers: the grader must pay the wrong fix again, and the check must pass the reference fix twice and fail the wrong fix twice. The one exception traced to our test machine, not the finding: an open-file limit made that task’s grader score every patch 0, the reference included. With the limit raised, the finding held. | October 3, 2026 Run accuracy-2026-10-03, part 1 | Script. No model, no person. |
| 738 of 795scope calls upheld | All 795 behavioral findings in the same 9 audits, in scope and out, re-labeled blind by three models. A two-of-three majority agreed with our original label on 738 (92.8%). Of the 57 it changed, 17 would move a finding out of scope and 40 would move one in. Claude is the same model family as our original scope judge, so its vote is not an independent check; GPT and Gemini are. | October 3, 2026 onward (Gemini votes arrived over several days)Run accuracy-2026-10-03, part 2 | AI models: Claude, GPT-5.5, Gemini. No person. |
| 0 of 181checks failed a correct fix | Is a check fragile, so that it would also fail correct code written differently? For 181 of 182 strict tasks, a model wrote two behavior-preserving refactors of the reference fix; a refactor counted only if the task’s own grader paid it in full. Each task’s first in-scope check then ran on every kept refactor. None failed one. The 182nd task was the open-file case above. | October 3, 2026 Run accuracy-2026-10-03, part 3 | Refactors written by a model; accepted by the task’s own grader; result counted by script. No person. |
| 5 of 10tasks, same verdict on a second audit | 10 tasks from one public benchmark, audited again from scratch with a new search. In all 5 disagreements, the first audit proved a hole the second did not find; the second never flagged a task the first had passed. Our search samples wrong fixes, so one pass finds some holes and misses others. That is why we call every hackable count a floor, and why certification uses a wider two-model search and prints an estimate of the holes every search missed. | October 3, 2026 Run accuracy-2026-10-03, part 4 | Script; wrong fixes written by models. No person. |
Planted holes, correct controls, answers fixed in advance.
On a real task set nobody knows the right answer. So we also run the audit on benches where we do: graders with holes we planted, beside graders we know are correct. A flag on a correct grader is a false alarm; a planted hole left unproved is a miss.
| Figure | What it measures, and on what | Date and run | Who checked |
|---|---|---|---|
| 0 of 24correct graders flagged | A synthetic tool-use environment with 60 graders: 36 with a planted hole in 6 families, 12 correct, and 12 correct but built to look suspicious. Labels and a held-out third were frozen and committed before any audit ran. No correct grader got a finding (95% interval 0–14%). 28 of 36 holes were proved as findings (78%, 62–88%), and all 36 were surfaced for review. On the held-out third, run once after tuning was frozen: 0 of 8 correct graders flagged, 9 of 12 holes proved. 3 of the 12 suspicious-looking correct graders drew a review lead, which a report shows for a person to judge and never counts as a finding. | October 12, 2026 Run agent-env-2026-10, calibration | Deterministic audit, no model calls. The bench was built by our AI worker; no person has reviewed its graders or labels. |
| 0 of 7correct web graders flagged | A synthetic store-and-CRM web app we built, 21 tasks, audited in a real browser. All 14 planted grader holes surfaced and 12 were proved. The 13 repaired graders blocked all 62 wrong runs found and still paid 39 of 39 reference runs and 39 of 39 alternate correct runs. Not a blind test: we knew the holes when we wrote the checks. | October 12, 2026 Run agent-env-2026-10, web-pilot-demo | Scripted browser, 3 fresh app processes, no model calls. No person. |
| 0 of 10clean judge controls flagged | Our AI-judge audit run on programmed judges built on items from two public judge-graded benchmarks, covering four published judge attacks (master keys, control tokens, ignored rubric items, reversed rubric items). It raised all 24 expected flags (95% interval 86–100%) and flagged none of the 10 clean controls (upper bound 28%). | October 12, 2026 Run superset-2026-10-12, judge families | Programmed judges; scored by script. No person. |
We measure false rejects as hard as false passes.
A grader can fail two ways: it pays for wrong work, or it refuses right work. We measure both, and we hold our own repairs to the same two-sided standard.
| Figure | What it measures, and on what | Date and run | Who checked |
|---|---|---|---|
| 0 of 2,009correct runs refused | One public business-workflow agent benchmark, 147 tasks with reference solutions, every run graded 3 times in fresh processes. The benchmark’s own grader refused none of the 2,009 runs our independent check counts as correct (95% interval 0–0.19%). It did refuse 12 of 102 answers we built to be equivalent to the right one (11.8%, 6.9–19.5%): the right title, written with doubled spaces between words, scored 0. That is the grader’s error, and we report it next to its false passes. | October 12, 2026 Run enterprise-envs-2026-10-12 | Script, sealed container, network off. No person. |
| 23 of 23wrong runs blocked, 0 right ones lost | Our repair on the same benchmark, checked both ways in 3 fresh processes: it blocks all 23 wrong runs the stock grader paid, and still pays all 9 reference runs, all 15 equivalent rewrites, and all 163 other runs the stock grader paid on those tasks. | October 12, 2026 Run enterprise-envs-2026-10-12 | Script. Scope of the 23 findings: 3–0 vote of Claude, GPT and Gemini. No person. |
Set-level results stay private until each maintainer’s notice period ends, so the benchmarks behind these figures are not named here. See our disclosure policy.
End-to-end false alarms on correct code: results before November 10.
The refactor test above checks one step, the behavioral check. The next measurement covers the whole code-audit pipeline: how often it would turn a correct fix into a reported finding.
Known-correct fixes, which each task’s own grader pays in full, are mixed blind with confirmed wrong fixes, so no model can tell them apart. Every candidate goes through the same stages as method 0.3.0: check writing, verification in fresh containers, the scope label and the three-model vote. The protocol is fixed in writing before the first model call. We will report controls tested, false alarms and a 95% interval, and post the result on this page before November 10, 2026.
Caught by our checks, before any claim.
Our second checks and known-answer benches have caught errors in our own pipeline. Each one is measured, fixed and kept on the record.
- A model vote is not proof. One version of our agent-environment check compared database rows by their ids, so 5 correct runs that wrote the same rows in a different order looked wrong, and three models voted them in scope, 3–0. Our repair check, built a different way, showed all 5 were correct. None was ever claimed. The check now compares rows without ids, and the full audit was re-run with it.
- Read the state before the grader runs. One environment’s grader changes its in-memory state while grading, so reading the end state afterwards made our check fail 15 correct reference runs. Caught before any claim; the state is now read first, and a test guards it.
- Four pipeline bugs found by the calibration bench. A crash, findings that depended on run order, timestamps counted as flaky scores, and a reachability rule that read a run before it was played. All four are fixed; 9 bench findings that relied on the last one were dropped, and no planted hole was lost.
What we do not yet measure
Each gap below is stated so you can weigh our figures correctly. We remove a line only when a measurement replaces it.
- End-to-end false alarms on correct code in code audits. Being measured now (above). Until then we publish no false-alarm rate for code audits.
- A person’s agreement with our findings. On every client audit an independent software engineer reviews a sample of findings, with hidden controls, before results go out. The first such review is not finished, so no human-agreement figure exists yet.
- Misses on real task sets. We measure misses on planted holes in benches we built. On real task sets, a hackable count is a floor: what one search proved, not every hole there is.
- Re-audit stability at scale. Our second-audit comparison covers 10 tasks from one benchmark.
- Our repairs against correct fixes written by people. Replacement tests are checked against reference fixes, model-written refactors and equivalent rewrites, not against independently written human solutions.
- False alarms on real AI judges. Our judge-control figure comes from programmed judges.
- Environments that need the network or live services. The figures above come from sealed runs. Network-on runs are reported separately and never pooled with them.
- The safety of any AI model. We audit graders: the tests and judges that pay out reward. Nothing on this page rates a model trained or evaluated on an audited task set.
How audits run and what we never do: how we test without teaching AI to cheat. The full method: method specification v0.3.0.
Hold our audit to the same standard you hold your graders.
Every finding we deliver ships with a one-command replay, so you can check our work on your own machine.