A certificate for task sets whose tests hold up.
Plumbline Verified covers one named, versioned release of a task set. We audit it, every hole we confirm is closed with verified tests, and a separate re-audit of the repaired release must find too few holes to matter, by a rule published here before any audit starts.
Certified releases
| Release | Vendor | Issued | Re-audit | 95% upper bound |
|---|---|---|---|---|
| No certificates issued yet.Criteria v1.0 take effect on November 10, 2026. Each certificate will be listed here with its figures and a badge that links to its entry. | ||||
Six criteria. All must hold.
A false pass is a solution the task’s tests accept although it does not do what the task asks, confirmed by a separate check in the task’s own sealed container. A task is hackable when it has at least one false pass the task as written rules out.
- Audit
We audit a random sample of at least 50 tasks of the release, or every task if it has fewer than 50.
- Repair
Every false pass the audit confirms is closed with replacement tests, verified in the task’s container: the reference solution still passes, and every known wrong solution now fails. The vendor adopts them in the release, or writes its own, verified the same way.
- Re-audit
A fresh audit of the repaired release, on a new random sample, finds few enough hackable tasks that the 95% upper bound on the rate (Wilson interval) is under 10%.
50 TASKSnone100 TASKSat most 4200 TASKSat most 11A release of fewer than 35 tasks cannot meet that bound even with none found, so it is re-audited in full and certified only if no task is hackable. Its certificate still prints the bound.
- Review
AI models from three companies each judge every finding blind. The majority decides, and their agreement is printed on the certificate.
- Wider search
The re-audit attacks each task requirement by requirement and from known hole patterns, with wrong solutions written by attacker models from two companies. The certificate prints an estimate of the hackable tasks every search missed, beside the confirmed count and never in its place.
- Independence
The re-audit is its own run, never one that wrote or verified any of the release’s replacement tests. Fees are fixed and never tied to the outcome, the method is fixed before the audit starts, any tests we wrote are disclosed, and a human reviewer who did not run the audit must check a random sample of findings, with hidden controls, before any certificate is issued. No certificate has been issued yet.
What it says, and what it doesn’t.
A certificate states
- The release, its version, the vendor and the date issued
- The criteria and method versions, with the run records behind every number
- For the first audit: tasks audited, false passes found, and how many were repaired and by whom
- For the re-audit: tasks audited, hackable tasks, the 95% upper bound, and the estimate of what the searches missed
- How far the AI models from three companies agreed, and how far the required human reviewer agreed
- How many tasks’ tests we wrote or verified, if any
A certificate does not
- Guarantee that no task in the release can be gamed. Every search misses something; the certificate estimates how much
- Cover tasks outside the sample, except through the rate and its interval
- Cover any other release. A new version needs a new certificate
- Cover behavior the task never asked for, or the harness, containers and scoring code outside the task’s tests
- Grade tasks for difficulty, usefulness or contamination
How long a certificate lasts
A certificate covers one named release and is valid for 12 months from the date issued, as long as that release’s tests do not change.
- Updates during the year. With Keep Current, we audit every new or changed task in each update to the same criteria. When they pass, the certificate moves to the new version with the same expiry date. If more than a quarter of the tasks are new or changed, the release needs a full re-audit.
- After 12 months. Renewal is a fresh re-audit on a new sample, under the criteria in force at the time. A certificate is always judged by the criteria version printed on it; publishing a new version of the criteria never cancels an existing certificate.
- A live badge. We serve the badge, and it shows the release and its expiry date. It changes to “expired” or “superseded” by itself if the certificate lapses or the vendor ships a version it does not cover.
- Errors. If we find a certificate was issued in error, we withdraw it and the registry says why.
Failing
A release that misses any criterion gets no certificate. We never issue a partial or softened one. The vendor may repair the release and ask for a new re-audit, on a new sample. Run records are kept for at least three years so anyone can re-check a certificate.
One fixed fee. Pass or fail.
The re-audit to the published criteria is a fixed fee per release, quoted up front from the release’s size. The fee is the same whether the release passes or fails, and so is the fee to keep a certificate current.
The badge
Each certified release gets a badge that links to its registry entry. The badge shows the release and its expiry date, and updates itself if the certificate lapses.
<a href="https://plumblinegrader.com/verified#RELEASE"><img src="https://plumblinegrader.com/verified/badges/RELEASE.svg" alt="Plumbline Verified"></a>For website owners: paste this where you list your release. Page updated October 8, 2026.