Grading a coding agent on agent-building work: reading pass rates

You asked a coding agent to build a customer-service agent. The first thing to grade is whether the result survives unseen work, not whether the transcript looks plausible. A demo is a highlight reel; a scorecard is a lab notebook.

Coding agent evaluation gets harder when the deliverable is another agent. The builder can show you a neat interface, a plausible tool call, and a clean transcript. None of that proves the agent will hold up on tasks it has not seen.

Demos hide construction failures

Before you trust a coding agent to build an AI agent, put the result against a benchmark that hides the training examples. The test should be messy enough to catch shortcuts. The usual failure is grading the builder on the tasks it helped write. If the agent can see the test, the pass rate measures memorization, not construction.

hyper-tau-bench is one concrete setup for this check. The arXiv identifier 2609.04611 corresponds to a paper titled τ^τ-Bench: An Environment for End-To-End, Realistic Agent Construction. The benchmark contains 53 tasks: 6 airline_plus, 6 retail_plus, 6 telecom, and 35 banking_knowledge. In it, a submitted agent is evaluated on unseen customer-service tasks through the τ³-bench inner loop, and the developer receives a reward equal to the agent's pass rate.

Pass rates need an expert ceiling

Pass rate is the metric that survives contact with real work. It tells you how often the agent completes the task, not how charming the transcript is. Compare it against an expert ceiling: if a human can finish the task reliably, the agent's score should be judged against that bar, not against a demo.

The bar matters most when the agent handles records, not a toy chatbot. Read cost with the score: high spend and a low pass rate point to looping, low spend and a low pass rate point to early stopping, and a high pass rate on a fragile harness points to a test that was too easy.

The harness is part of the measurement

A reproducible harness setup turns a one-off run into a repeatable evaluation. Keep the same task set, keys, and limits for every run.

• Pick unseen customer-service tasks first. Done looks like a task list the builder has not seen, with records, API calls, and inherited code.
• Wire the real environment. For hyper-tau-bench release tasks, set OPENAI_API_KEY and OPENROUTER_API_KEY. Add ANTHROPIC_API_KEY only when claude-code is the Developer harness. The environment is ready when the required keys are set.
• Cap the cost before the run. Set a budget ceiling and stop the run when it is hit. Log the spend number next to the pass rate.
• Score against an expert ceiling. Run a human or a known-good baseline on the same tasks, then compare the agent's pass rate to that ceiling. Finish with a gap, not a raw score.
• Reproduce the harness. Log the harness choice, task set, keys, and cost cap so another engineer can rerun the same test. In hyper-tau-bench, the Developer seat can use codex as the default harness, or claude-code, opencode, or prime-agent. The run is reproducible when a teammate can rerun it from the log.

Auditable runs separate failure modes

When the run finishes, write down what the agent changed, what it broke, and what it could not do. A useful report includes the task list, the harness, the spend log, and the scoring gap. If a teammate cannot rerun the test from that report, the evaluation is a story, not a measurement.

Use the scorecard to separate three failures. The builder may be bad at planning, bad at tool use, or bad at finishing. Keep the notes next to the score so the pattern is visible across runs before you scale it.

For a customer service agent, the highest-value checks are the boring ones: can it read the record, can it call the right API, can it update the state, and can it stop when the task is done? Record which failure mode appears, then rerun the same harness before you change the model.