Three free audits. Certification grade.
For our launch, three teams get our most thorough audit, free: one delivery or environment set, up to 100 tasks, every wrong fix the grader pays for proved, and replacement tests that close the gaps. Plumbline Verified is on the table for a release that meets the criteria. It is earned, never given.
3 places · $0 · disclosed as part of the launch partner program
The people who build, buy and vouch for training tasks.
In our audits of 11 public task sets, 190 of 487 testable tasks paid full reward for a wrong fix in one search pass: about 4 in 10. A partner audit runs a deeper search than that, on your own tasks.
Check a delivery before it trains
Audit a vendor delivery you have already received, or a set you built, before it reaches a training run.
Acceptance criteria you can measure
Turn “is this batch good enough?” into a count of paid wrong fixes, with the proof for each one.
Proof you can show a buyer
An outside audit of one release, and Plumbline Verified if it meets the criteria after any fixes.
We aim for one of each. We choose for fit, not for who applies first.
The audit a certificate would rest on.
Up to 100 tasks from one delivery or environment set. Each task runs through its own grader, unchanged, in a sealed container: a fresh container per run, network off.
- 01 · DEEP SEARCH
Wrong fixes, per requirement
AI models from two companies write a plausible wrong fix for each requirement the task states, plus attacks drawn from our library of known test gaps. The audit reports what it confirmed and an estimate of what a search can still miss.
- 02 · PROOF
Every finding proved
A finding counts only when a separate behavioral check passes on the reference fix, fails on the unfixed code and fails on the wrong fix, again in fresh containers. Each one ships with a one-command reproduction: bash and Docker, nothing of ours to install.
- 03 · TWO-SIDED REPAIR
Replacement tests, measured both ways
For every gap, tests that fail the wrong fixes we found and still pass correct ones. We measure both sides and report both numbers.
- 04 · PERSON IN THE LOOP
A human-reviewed sample
A person who did not run the audit reviews a random sample of findings with hidden controls mixed in, and we report their agreement. Model reviews are labelled as model reviews.
You also get the evidence bundle (run logs and their hashes, the method version and the run manifest) and a walkthrough with your team.
Three things in return.
- Written feedback.What was useful, what was wrong, what you would change. We publish none of it without your permission.
- Permission to describe the engagement.Named or anonymous, your choice, in words you approve before anything goes out.
- For vendors: Verified, if earned.If the release meets the criteria, it carries Plumbline Verified, disclosed as issued under the launch partner program.
Free does not mean friendly.
- A certificate rests on a fresh audit.Run after the release is final, on a fresh random sample, never by the run that wrote or checked any of its tests.
- Failed means failed.A release that misses the criteria gets no certificate, never a softened one. You may keep the result private.
- The fee is zero either way.Pass or fail, a partner pays nothing, and no fee is ever tied to the number of findings.
- Disclosure on the certificate.If we wrote or checked tests in the release, the certificate says so, with the number of tasks.
- A person checks the sample.No certificate is issued until a reviewer who did not run the audit has checked a random sample with hidden controls.
These are rules 1, 3, 4, 6 and 7 of our independence policy, measured against the published Verified criteria. No certificate has been issued yet.
On our machines, or on yours.
Your tasks are never published, and we sign your NDA before anything changes hands.
Send the tasks
Container images or Dockerfiles, reference fixes, tests and grading scripts, the way your environment runs them. Harbor and SWE-bench-style tasks run as they are; for a custom format, one example task tells us what we need.
Keep the tasks inside
The audit engine needs Docker and access to AI model APIs. We can run it on machines you control, and you choose which model providers may see task content. Running with only your own model endpoints is in development: ask us where it stands.
Apply for one of three places.
Two minutes. Tell us what you would audit and where it should run.
I read and answer every application myself. The audit engine is built and run with AI agents, and every finding ships with a one-command reproduction, so your team can check the work instead of taking our word for it.
Launch partner application
Your organization, what you would audit, and where it should run.
Before you apply
Why is it free?
Because three well-run audits, on real deliveries, are worth more to a new company than three invoices. We want the feedback and, with your permission, the chance to describe the work.
Does a partner get an easier audit, or an easier certificate?
No. Same method, same criteria, same rules. A release that fails does not get Plumbline Verified, and the fee is zero whether it passes or fails.
Will you publish our results?
Not without your permission. Your tasks and your results stay private. You decide whether the engagement is described at all, and whether by name or anonymously.
Our public task set is in your Index. Can we still apply?
Yes. Index numbers come from fixed public runs and are never changed, delayed or withheld for a partner or a customer, and any rated company that becomes a partner is listed as one.
What happens to our tasks and code?
They run in sealed containers with the network off. Task text and code go to AI model APIs to write wrong fixes and checks; you choose which providers may see them. Or we run the audit on machines you control.
How long does it take?
We agree a delivery date with you before we start, once we have seen the task format and the access we will have.
What happens after the pilot?
Nothing you have not agreed to. If you want more audits, our prices are on the pricing page. There is no automatic renewal and no obligation.