Harness Scorecard

The Harness Scorecard

One formula and eleven metrics, so two harnesses can be compared by something other than adjectives. The primary unit is cost per correctly verified outcome — not successful runs, which is a self-report and the one thing a broken harness is still good at producing.

Version 1.0 · Updated 2026-08-27 · 11 metrics

JSONMachine-readable

The unit

CVO = total cost of the run set / outcomes that passed independent verification

Numerator
Every cost attributable to the run set: inference, tools, infrastructure, and the human time spent approving, correcting or escalating. Failed and abandoned attempts are in the numerator. They cost money and they happened.
Denominator
Only outcomes that passed a check which does not share a failure mode with the agent that produced them. Runs that finished do not count. Runs the agent declared successful do not count.
What makes a verifier independent
A verifier is independent when it can fail on inputs the producer succeeds on. A rule, a test, a second system or a person all qualify. The same model prompted to grade its own output does not: it inherits the blind spots that produced the error.
Why not successful runs
Successful runs is a self-report, and it is the number every vendor already publishes. It counts the agent finishing, which is the one thing a broken harness is still good at. Cost per correctly verified outcome refuses to move until something outside the agent agrees that the work is done.

HSC-01 · Cost per correctly verified outcome

What this is, and what it is not

This is a proposal, not a measured result. Nothing here has been run against a real system, no threshold is calibrated, and the eleven metrics were chosen by argument rather than by finding out which of them predict anything. The file says as much in its own evidence block: theoretical, low confidence.

It is published so it can be argued with. The claim it rests on states what would falsify it, and the day a measurement contradicts it, the measurement wins.

theoreticallow

The claim this rests on: HE-CLAIM-005 — retrievable over MCP with get_claim.

The metrics

Outcome

HSC-01

Cost per correctly verified outcome

currency/outcomeBetter when lower

Every cost attributable to a run set — inference, tools, infrastructure and the human time spent approving, correcting or escalating — divided by the number of outcomes that passed an independent check.

How to measure it

Fix the task set and the verifier before running anything. Sum the cost of the whole set, failed attempts included. Divide by outcomes the verifier passed, not by runs that finished.

How to fake it

Narrow what counts as an outcome, or let the agent's own report stand as verification. Both shrink the denominator's honesty rather than the numerator's cost.

Read offACM-11

HSC-02

Verified yield rate

percentBetter when higher

The share of attempted tasks that produce an outcome passing independent verification.

How to measure it

Attempts in the denominator, verified outcomes in the numerator. An attempt abandoned by a timeout is still an attempt, and a run the agent gave up on is still an attempt.

How to fake it

Attempt only the tasks the harness already handles. A yield rate with no declared task set is a number about the task set, not about the harness.

Read offACM-11

HSC-03

Verification gap

percentage pointsBetter when lower

The self-reported success rate minus the independently verified one, in percentage points. It measures whether the harness can tell that it succeeded.

How to measure it

Record what the agent claimed for each run and what the verifier found, then subtract. Report the sign: a harness that underclaims has a different problem from one that overclaims, not a smaller one.

How to fake it

Stop asking the agent to self-report. The gap disappears and so does the only signal that the harness cannot distinguish success from failure.

Read offACM-10ACM-11

Cost and effort

HSC-04

Tokens per verified outcome

tokens/outcomeBetter when lower

Model tokens, input and output, across every attempt, divided by verified outcomes. The cost figure that does not move when a vendor changes its price list.

How to measure it

Count tokens for the whole run set — retries and abandoned attempts included. Report the model and its version alongside; without them the number compares nothing.

How to fake it

Move work out of the model and into an uncounted tool or a longer human review. The tokens fall and the actual cost does not.

HSC-05

Human interventions per verified outcome

interventions/outcomeBetter when lower

Approvals, escalations and manual corrections per verified outcome. Not whether a person is in the loop, but how often the loop closes on one.

How to measure it

Count every event that required a person to act before the outcome could stand. Rubber-stamped approvals count: what is measured is the demand placed on people, not the quality of their attention.

How to fake it

Remove the gates. The rate falls to zero and nothing about the harness improved — which is why this is read next to the verification gap and never alone.

Read offACM-08ACM-09

HSC-06

Time to verified outcome

secondsBetter when lower

Wall-clock time from a task being accepted to its outcome passing verification, including the wait in front of any human step.

How to measure it

Report the median and the 95th percentile, or do not report it: a long tail of stuck runs is exactly what a mean hides.

How to fake it

Measure to the agent finishing rather than to verification passing. The queue in front of a human reviewer is where the hours actually are.

Read offACM-07

Reliability

HSC-07

Determinism band

percentBetter when higher

The share of the evaluation set that reaches the same verified verdict across repeated runs of the same input, with the harness unchanged.

How to measure it

Run the whole set at least five times and declare N. A case is inside the band when every repeat lands on the same verdict — not when the average is good.

How to fake it

Run once and report the score. A single-run number contains this metric's answer and hides it.

Read offACM-11

HSC-08

Recovery rate

percentBetter when higher

Of the runs that hit a failure — a tool error, a refusal, a timeout, a rejected plan — the share that still reach a verified outcome.

How to measure it

The failure has to be visible in the trace before it can be counted, so this cannot be computed without a correlated run trace. Never having failed is not recovery and must not be counted as it.

How to fake it

Suppress the failures instead of recovering from them. A retry loop that swallows a tool error raises this number and the verification gap at the same time.

Read offACM-10

HSC-09

Containment ratio

percentBetter when higher

The share of the harness's write-capable, irreversible or regulated actions that sit behind a control which is both feedforward and deterministic.

How to measure it

List the actions from the tool catalogue, name the control that stops each one, and read that control's two axes off the Agentic Control Matrix. Gates decided by judgement do not count here — the human intervention rate counts those.

How to fake it

Reclassify actions as reversible. The ratio rises and the blast radius does not move.

Read offACM-01ACM-03ACM-17

Evidence

HSC-10

Reconstructability

percentBetter when higher

Of a random sample of past runs, the share that can be reconstructed end to end from the log alone: inputs, model, tool calls, outcome.

How to measure it

Sample runs nobody chose, and reconstruct them without asking the team and without the original operator present. A run that needs a person to explain it did not pass.

How to fake it

Sample the runs already known to be clean — or log everything, secrets included. This is read together with whether the log holds data it never needed.

Read offACM-10

HSC-11

Regression catch rate

percentBetter when higher

Of defects deliberately planted in the system, the share the evaluation suite catches before release.

How to measure it

Plant the kinds of defect you actually fear, one at a time, and run the suite on each. A suite never seen failing has an unknown catch rate, not a perfect one.

How to fake it

Plant the defects the suite was written from. The rate goes to 100% and measures nothing but the author's memory.

Read offACM-11

What a result has to state

A number without these is not comparable to another number. HarnessBench reports all six or reports nothing.

  • modelThe model and its exact version
  • task_setThe task set, and where it came from
  • verifierThe verifier, and why it is independent of the agent
  • repeatsHow many times the set was run
  • dateThe date it was run
  • cost_basisWhat the cost figure includes, human time included