Private notice first, then the public record.
Plumbline audits the graders in public benchmarks, task sets and evaluation tools. When we find a grader that pays full reward for wrong work, we tell the people who maintain it first, privately, with the evidence and, wherever we can build one, a tested fix. We go public only after they have had time to act. Last updated October 15, 2026.
1. Private notice first
- Who gets it. The maintainers, through the private channel the project itself publishes: a security contact or
SECURITY.mdif there is one, otherwise the contact address in the project’s own README or paper. We never open a public issue or pull request first. - How it is sent. By email, sent personally by our founder. Notices never go through automated sequences.
- What it contains. What we found, with file and line at the exact commit we audited; how many tasks it affects and how we counted them; how we checked: the project’s own grader, run unchanged, in fresh sealed containers, repeated; where we can build one, a fix tested both ways, so it rejects the wrong work we found and still accepts the reference solutions; and a script that reproduces each case from the project’s pinned commit with bash and Docker.
- Who checked what. We say which checks were done by AI models, such as a three-model scope vote, and never describe a check as done by a person unless a person did it.
- What it never contains. No prices, offers, sales pitch, meeting links or marketing. A notice is a notice, and follow-ups carry the same rule.
2. The hold
- At least 30 days. We do not discuss a finding publicly, or name the project in connection with it, until at least 30 days after the notice. The notice states the date.
- More time on request. Every notice invites the maintainers to tell us if anything is wrong or if they need more time. We extend the hold for a good-faith reason, such as a fix in progress.
- Earlier, if the maintainers choose. If the maintainers ask us to raise a finding publicly sooner, for example as public issues in their repository, we do that for the items they name, and only those.
- Findings from before our launch. Findings from notices sent before our public launch on November 10, 2026 stay private at least until their hold ends, unless the maintainers agree otherwise.
3. Credit for fixes
When maintainers fix a problem, we credit the fix when we publish, and report the finding against the commit we audited, with the fix noted. We report what we measured, and we never describe a fixed problem as still open.
4. What we publish
When the hold ends we may publish the finding, the counts, the evidence and the fix. Every number comes with its scope: which commit, how many tasks, how they were sampled. We correct any error the maintainers show us.
Right of reply. Any maintainer or vendor named in something we publish can respond, and we publish the reply alongside our findings.
5. Client work is different
A client’s own tasks and results are never published. If an engagement touches a public task set, our own public research on that set follows this policy and uses only public material.
6. No fee, no conditions
A notice never comes with a price, an offer or a condition. Whether a project fixes a problem, replies, or ever does business with us does not change what we find or publish about public material. No fee is ever tied to the number of findings.
7. Contact
Questions about a notice you received go to the founder directly: shane@plumblinegrader.com. We reply within one business day. To report a security problem in our own site or tools, see Report a vulnerability.