Score coding agents on private tasks, then price the harness

You can compare vendor claims and still buy the wrong coding agent. In the September 2026 Real-SWE leaderboard, the top listed result was 38.8% for Fable 5.1 running in Claude Code. The second listed result was 33.8% for GPT-6 Astra running in Codex CLI. Before the demo, run a small private-repo first-attempt test and price the harness per run. Scorecard: pass@1 on a small set of tasks from your own private repo, and the harness's added cost and success delta per run versus a minimal harness.

Leaderboards narrow the field; your repo decides

Public leaderboards help you shortlist, but they do not show how a candidate behaves inside your code, your tests, and your review habits. The Real-SWE benchmark tests model-and-harness pairs with native harnesses, not models by themselves.

Its tasks ask agents to operate inside proprietary systems that are not exposed on the public internet. It runs each agent in an isolated sandbox, uses Harbor-format tasks, and injects verifiers when grading.

Real-SWE's resolution rate is pass@1, computed as the average of eight independent runs for each task. Read the leaderboard numbers as a sanity check. The spread shows that pair choice can move the outcome, but your private tasks will move it further. A public score can tell you whether a pair is plausible. It cannot tell you whether it will survive your lint rules, your test fixtures, your branch protection, or your release checklist.

Run a private-repo scorecard before any demo

Pick a small set of private tasks that are boring, real, and gradeable. You want tasks where a wrong answer is obvious: a failing test, a missing endpoint, a broken migration, a security check that should pass. Avoid tasks that require long context, hidden tribal knowledge, or a human to interpret the result. Run this checklist before you shortlist.

• Run the same private task on the candidate agent and a minimal harness, then record whether the first attempt passes.
• Time the run and note where the agent stalls: setup, tests, review, or commit.
• Compare the harness's added cost per run against the minimal harness for the same task.
• Check whether the agent executes its code and whether that execution is safe in your environment.
• Score the diff for human review burden: does it need a rewrite, a patch, or a merge?

Each item catches a failure mode that a demo hides: first-attempt pass rate shows whether the agent can close the loop without hand-holding; stall time exposes where the harness adds friction; cost delta reveals whether the wrapper is worth its price; execution safety determines whether the agent can be trusted near real work; review burden separates a starting point from a liability.

Greptile, Cursor, and Devin are aligned on the idea that agents should execute their code. Execution is where the harness earns its keep. A wrapper that only edits text is less useful than one that can run tests, inspect failures, and repair the patch. Your private tasks should force that behavior.

Price the harness as a separate line item

Most vendor pitches bundle the model, the interface, the sandbox, the memory, the integrations, and the support into one number. That number hides the part you should be negotiating: the harness. If the harness adds cost without success, treat it as a fee.

Use the same private task in the full harness and in a minimal setup. Record the success delta and the price difference per run. When the full harness wins by a small margin, ask whether that margin is worth the extra spend. When it loses, the model may be fine and the wrapper may be the problem.

The harness cost is the line item most buyers forget. A strong model can look weak inside a clumsy harness. A weak model can look acceptable inside a harness that does the right things at the right time.

When the vendor asks for a pilot, hand them your scorecard. Ask for the same private tasks, the same minimal harness, and the same pass@1 measurement. If they object, ask what they will change before the pilot starts.