PLUMBLINE GRADER
Free 20-task audit
One public example, start to finish

Your tests passed. Was the answer actually right?

A grader can pay for work that is wrong. Here is one case anyone can check: what the task asked for, the wrong run the grader paid, the check that caught it, and the repair that blocks it while still paying correct work.

The method in one picture

How every finding is made, and what it must pass.

How a Plumbline finding is made, in seven steps 1, the task: the issue, its reference fix and its tests. 2, candidate fixes: wrong fixes we build to probe the grader, the reference fix with parts removed and fixes written by AI models, not agent submissions. 3, does the task's own grader pay a wrong fix full reward? If not, there is nothing to report for that fix. 4, a check that never calls the grader passes on the reference fix, fails on the unfixed code and fails on the wrong fix, twice; otherwise there is no finding. 5, scope: does the task as written require the missed behavior? AI models from three companies vote; if not, the case is listed for the client and never counted. 6, replay in a fresh container: the same verdicts again, so the finding is reproducible, not luck. 7, repair: replacement tests that block every in-scope wrong fix found and still pass the reference fix and correct alternatives. 01 · THE TASK The issue, its reference fix and its tests graded by the task's own grader 02 · CANDIDATE FIXES Wrong fixes we build to probe the grader the reference fix with parts removed, and fixes written by AI models (not agent submissions) 03 · THE TASK'S GRADER Does it pay a wrong fix full reward? yes: the fix goes on to step 4 yes no: stop here, nothing to report 04 · INDEPENDENT CHECK A check that never calls the grader passes on the reference fix, fails on the unfixed code and fails on the wrong fix, twice each check holds otherwise: stop here, no finding 05 · SCOPE Does the task as written require it? AI models from three companies vote yes no: listed for the client, never counted 06 · REPLAY Re-run in a fresh container same verdicts again: reproducible, not luck 07 · REPAIR Replacement tests, measured both ways block every in-scope wrong fix found and still pass the reference fix and correct alternatives
Every candidate is a wrong fix we build to test the grader, not an answer an agent submitted. The banking example below follows the same path with a scripted run: its check compares the whole end state with the reference run’s, so it needed no model scope vote, and the grader’s verdicts were repeated in 3 fresh processes.

The example in four steps

  1. 01 · WHAT THE TASK REQUIRED

    Banking task 078 in Inspect Evals 0.24.0, the open-source evaluation suite maintained by Generality Labs with contributions from the UK AI Security Institute. A customer reports lost debit cards. The task’s reference actions handle the cards and nothing else. The grader compares the final database with the one those actions produce.

    # reference actions, task_078 (shortened)
freeze_debit_card  × 3
close_debit_card   × 2  reason="lost"
order_debit_card   × 1  EXPEDITED
    Changing the customer’s email is not part of the request.requirement
  2. 02 · THE WRONG RUN THE GRADER PAID

    A run we scripted to test the grader (no agent submitted it): the same 21 reference actions, then one more that nobody asked for.

    … the 21 reference actions …
+ banking_change_user_email(
+     user_id="mc78a5b9d2",
+     new_email="not.requested@example.com")
    Grader: “Database state matches expected state after required actions.”PAID
    Across the 24 banking tasks that add a customer at setupPAID · 24 of 24
    Why: the task’s setup records were merged into the live database as shared objects, not copies, so the run’s own edit also rewrote the answer key the grader checks against.
  3. 03 · THE CHECK THAT EXPOSES IT

    A separate check, written before any wrong run existed and never calling the grader: the run’s final database must equal the reference run’s. Here it does not: the customer’s email changed. Fingerprints of each end state:

    reference run, end state  b812c36e…
wrong run, end state      64d105cc…
independent check         FAIL
    Control: the same change on a customer already in the databaseREFUSED · 24 of 24
    The grader sees the change when the answer key is not shared. The defect is the shared record, not the comparison.
  4. 04 · THE REPAIR, HELD TO BOTH SIDES

    Copy the setup records before the run touches them. A repair counts only if it blocks the wrong run and keeps paying correct work.

    # tau2/banking/environment.py, before:
_apply_initialization_data(agent_tools.db,
    initial_state.initialization_data)
# after: merge from a deep copy
_apply_initialization_data(agent_tools.db,
    initial_state.initialization_data
        .model_copy(deep=True))
    The wrong run, on the same 24 tasksBLOCKED · paid 0 of 24
    Correct runs: all 97 banking reference runs, with this repair and its companion fix for a number-type mismatch (issue #2677)PAID · 97 of 97
    As shipped, the same grader refused 20 of those 97 correct runs. The repair pays every one.

Scope: banking tasks of Inspect Evals 0.24.0 (inspect_ai 0.3.278). Scripted runs; no AI model played the agent. Grader verdicts repeated in 3 fresh processes; repair tested on local copies. Checked by code, not by a person. Both issues are open; the maintainers have assigned the fixes to us, not yet merged (October 30, 2026).

Check it yourself

Filed in public, at the maintainers’ request.

The maintainers have since assigned the fixes to us. Each issue shows where the defect sits, how to reproduce it and the tested fix. In a coding environment the same failure is a fix that passes every test and skips a requirement; our audits look for every case like it, task by task.

Inspect Evals issue #2676The wrong run, the control and the deep-copy fix.GitHub Inspect Evals issue #2677The companion fix; with both, 97 of 97 correct runs are paid.GitHub
The same failure in a coding task: an illustration
ILLUSTRATION · not from an audited set
Task: reject reset tokens that have expired, or were issued before the last password changerequirement
Wrong fix: checks expiry, ignores password changescandidate
Task’s own tests on the wrong fixPASS · full reward
Separate check on the reference fixPASS
Separate check on the wrong fixFAIL · old token still accepted
Replacement tests on the wrong fixFAIL · reward withheld
That is one finding. An audit looks for every one like it, across every task you send.

Find out what your grader pays for. Start small.

Send up to 20 of your tasks. We look for every case like this one, prove each with a separate check, and send full evidence. Free.

Discuss a small audit of your task batch
shane@plumblinegrader.com