We audit the graders that train and rank AI models.
A coding environment pays a model when the tests pass. When the tests are weak, wrong code earns full reward, and a model in training learns to pass tests instead of solving tasks. Plumbline Grader finds the wrong fixes a grader accepts, proves each one, and writes the tests that close the gap.
What our audits of public task sets found
public task sets audited.
testable tasks paid full reward for a wrong fix in one search pass: about 4 in 10 pooled, from under 20% to 69% by set.
findings we re-ran from scratch reproduced.
findings now public as Inspect Evals issues, each with a tested fix.
Since October 2, 2026, the maintainers of at least 20 public benchmarks and evaluation suites have heard from us privately, with the evidence behind each finding, before anything is published. Aggregate results from our audits of public task sets. A task is testable when its reference fix passes its grader, an empty fix does not, and at least one wrong fix ran. Set-by-set results publish on November 10, after each maintainer’s private notice period ends.
We audit environments. We never build them.
An auditor that also sells environments has a reason to go easy on its own. We have none to sell, and we are paid the same whatever we find. These rules are published so you can hold us to them.
- We never build or sell environments. No environment of our own to favor, and no reason to go easy on yours. The teams that build environments are our customers, not our competitors.
- Flat fees. Priced per task, per month or per release, and fixed before we start. Never per finding, never tied to a result. Public prices, and how the rest are quoted, are on the pricing page.
- Index numbers are never for sale. Index numbers come from fixed public runs and are never changed, delayed or withheld for a customer, and any company whose task set appears in the Index and is also a customer is listed as one.
- Failed means failed. A release that misses the Plumbline Verified criteria gets no certificate, never a softened one. The fee is the same either way.
Evidence first. Then the conversation.
Each task runs through its own grader, unchanged, in a sealed container: a fresh container per run, network off. Nothing counts as a finding until a separate check shows the fix is wrong. See how an audit works.
Published before you buy
The method specification, the Plumbline Verified criteria and the independence policy are public in full. Read exactly what an audit is held to before you talk to us.
Re-run any finding yourself
Every finding ships with reproduce.sh: one command, bash and Docker. No Plumbline code, no API key, no model. You check the work instead of taking our word for it.
Fixes held to both sides
A replacement test must fail every wrong fix we found and still pass the reference fix. We verify both in the task’s own container, then attack the repaired tests again with fresh wrong fixes.
Private notice first
Findings on a public task set go to its maintainers privately, with the evidence and a tested fix, at least 30 days before anything is published. Notices carry no prices and no offers. Your own tasks and results are never published.
The public record: check our work.
Four of our findings on the UK AI Security Institute’s Inspect Evals are public GitHub issues, filed at the maintainers’ request on October 13, 2026. Each one shows how to reproduce the problem and offers a tested fix. Any maintainer or vendor named in something we publish can respond, and we publish the reply alongside our findings.
reproduce.sh and its guide: re-runs any finding we deliver.Download
Built by a clinician who learned that passing is not proof.
Shane Cope is a dentist and technology entrepreneur. He studied computer science and trained in cybersecurity before earning his DDS and MSD, and has since founded several businesses, including HIPAA-regulated software for dental practices built with AI coding agents.
Building that system showed him the problem Plumbline exists to solve: AI-written code can pass every test and still do the wrong thing. Where patient data is at stake, “the tests passed” is not proof, so he built a way to show exactly what the tests miss.
In clinical work, a result counts when someone else can check it. Plumbline holds every finding to the same standard: a separate check that shows the fix is wrong, a re-run from scratch, and a reproduction you can run yourself.
How the work gets done, plainly.
The audit engine is built and run with AI agents, under Shane’s direction. Agents do the searching; the checks below decide what counts, and we always say which checks a model did and which a person did:
- Re-runs from scratch. A sample of findings is re-run from scratch, with no model involved.
- AI models from three companies. In our code audits, Claude, GPT and Gemini vote on the scope of every finding, and a two-of-three majority decides. They also judge sampled findings blind. In our accuracy study the majority upheld 738 of 795 scope calls. They are models, and we say so.
- An independent software engineer on every client audit. Someone who did not run the audit reviews a sample of findings before results go out, with planted controls to keep that review honest.
- You. Every finding ships with a one-command reproduction, so the last check is yours.
Plumbline Grader
Plumbline Grader is operated by IROS Systems LLC, Indiana. Founder: Shane Cope, DDS, MSD. Questions about our method, our independence or a notice you received go to the founder directly, and get a reply within one business day.
shane@plumblinegrader.com
Send 20 tasks. See what passes that shouldn’t.
The free sample uses the same deep search as a paid audit and comes with full evidence. No cost, no obligation.