Your tests passed. Was the answer actually right?
A grader that pays for wrong work teaches a model to pass instead of solve. We find the wrong answers your grader rewards, show the evidence, and deliver tests that reject them.
Built for RL labs, teams that build or buy RL environments, and teams that rely on AI judges.
In 11 public coding task sets, 190 of 487 testable tasks paid full reward for a wrong fix our search built, in one search pass.
$0, full evidence. The free sample is reviewed by AI models; every paid audit includes an independent engineer’s review of a sample.
A pass is not proof. A check is.
- 01
Build
Wrong fixes on purpose: trimmed reference fixes and AI-written bugs.
- 02
Grade
Each one runs through the task’s own tests in a fresh, sealed container, network off; any reward steps after the tests must pay it too. Only full-reward candidates advance.
- 03
Prove
A separate check passes the reference fix and fails the wrong one, twice. Only then is it a finding.
- 04
Replay
A fresh container gives the same verdicts. You get
reproduce.sh: bash and Docker, no model. - 05
Repair
New tests block every wrong fix found and pass the reference and tested correct alternatives.
The grader paid. The check said no.
Banking tasks in Inspect Evals 0.24.0, the open-source suite maintained by Generality Labs with contributions from the UK AI Security Institute. Filed in public at the maintainers’ request on October 13, 2026. The maintainers have since assigned the fixes to us.
We measure ourselves the way we measure graders.
known-correct fixes reported as findings (95% upper bound 2.5%).
known wrong fixes re-found, mixed blind with the correct ones.
sampled findings confirmed by an independent engineer, blind, with hidden controls.
findings gave the same verdicts when re-run from scratch.
Who checked: scripts and AI models, and for the 9 of 9, an independent software engineer we retained, working blind. The re-runs show a finding is stable; the engineer’s sample and the known-answer tests check whether it is right. All four figures come from public coding task sets: the 150 and the 50 from 75 tasks in 3 sets (October 15 and 17, 2026), the engineer’s sample from 4 sets, the re-runs from 9 audits. Full scope and intervals: our error rates.
Send 20 tasks. See what passes that shouldn’t.
We audit environments; we never sell them. Flat fees, never tied to findings.
Grader Audit: $40 per task, $1,500 minimum per batch, about 5 business days, independent engineer’s sample review included.