How we test without teaching AI to cheat.

• The models write text, nothing more. Our attacker models return candidate wrong fixes as text through their providers’ APIs. They have no tools and run nothing on our systems.
• A fresh, sealed box for every run. In code audits, every grader run, check and file read uses a new container made from the task’s image, removed when the run ends, so nothing one wrong fix does carries over to the next. Agent-environment audits grade each run in a fresh process inside a sealed container.
• Network off. Sealed containers start with no network at all. After start, our code inspects each container’s network setting and refuses to run unless it is off.
• Every capability dropped. All Linux capabilities are dropped and privilege escalation is blocked. Each container has memory, CPU and process limits. Where a task’s own verifier needs one capability back, it is added for that task only and recorded in the run manifest.
• When a task needs a network. Some agent environments cannot run without one, such as a browser that must reach the web app under test. Those containers share an internal network with no route to the internet. Where a task’s own harness needs outside access, a separate opt-in mode gives it one exit: a proxy that logs every request, with every capability still dropped. Those results are labelled and never pooled with sealed ones, and the mode is refused when we run inside a client’s own environment.
• Building is not grading. The network is on while a task’s image is built from its own recipe, and for one warm-up run with the reference fix to fill dependency caches. Grading in sealed mode is always offline.

The full container settings are in section 9 of the method specification, and how your own tasks are handled is on our security page.

• We train no AI models. Plumbline does not train or fine-tune models, on wrong fixes or on anything else.
• Not sold as training data. We do not sell or license the wrong fixes our search writes as training data. They go where evidence belongs: into the client’s report, or into a private notice to a public set’s maintainers, together with the tests that block them.
• Client tasks stay out of training. Your tasks are used only for your audit and never to train AI models.

• No cyber-offense, capture-the-flag or exploit-development environments. We audit graders in coding, tool-use, web and business-workflow tasks, and AI judges. We do not take on environments whose task is to attack a system.
• Our wrong fixes are ordinary code. A wrong fix is a code change that does the task badly and still passes a weak test: it skips a case, hard-codes an answer, or does only part of the job. That is the reward hacking a model in training would learn, caught before it can.

Our AI agents work under automatic safety checks. When one stops a step, we log the stop, skip the step, and never reword or retry it to get around the check. On October 12, 2026, for example, a check stopped two steps of a benchmark survey. Both stops are logged, neither was retried, and one benchmark was left out of that audit as a result.

Since October 2, 2026, the maintainers of 21 public benchmarks and evaluation suites have heard from us privately, with the evidence behind each finding and, wherever we could build one, a fix tested both ways. Each notice holds publication for at least 30 days. Four findings are public as GitHub issues because the maintainers asked us to file them sooner. Notices carry no prices and no offers, and owners are named only after their hold ends. The full rules are in our disclosure policy.

• We never sell environments, and never certify one we built. No product of our own to favor.
• Fees are never tied to findings. Flat fees, fixed before the work starts, the same whatever we find and whether a release passes or fails.
• We say who checked what. Every figure we publish says whether a script, an AI model or a person did the checking. See how we measure ourselves.

The rules in full: our independence policy.

Plumbline Grader is self-funded by its founder. As of October 15, 2026 we have received no outside investment, no grants and no model-provider credits. No AI lab, environment vendor or benchmark owner holds equity in Plumbline or funds our work. We have applied to startup credit programs run by Anthropic, Google and Microsoft; none has paid out. We use Claude, GPT and Gemini, and we pay for them. When any funding, grant or credit is received, we name it here.