Reading Vendor Benchmarks: 3 Questions, Astra's 99.9% vs 62.7%

Read a vendor benchmark as a claim about a specific runner, harness, and task, not as a general capability score. GPT-6 Astra made the gap visible: OpenAI highlighted a 99.9% ARC-AGI-3 result on its Provider Adapter harness, while ARC Prize's provider-neutral Standard harness produced 62.7% for the same model at maximum effort. Ask who ran it, which harness it used, and whether the task is closed-ended before you let the number move your decision.

OpenAI made Astra available on September 3-4, and its president, Greg Brockman, described the release as AGI. The higher score came with a lower reported cost: the Provider Adapter run cost $18,817, while the standard run cost $26,098. ARC Prize's policy allows two evaluation conditions and calls for each result to be labeled by the harness used.

The spread is the first clue

The spread shows where the runner, harness, and task changed. ARC Prize characterized ARC-AGI-3 as deterministic and closed-ended, and said it falls short of real-world complexity.

xAI's Grok 4.6 launch page is a second warning. It presented a ten-row evaluation table with four model columns and reported Grok 4.6 at 26% of tasks on Terminal-Bench v3.0, a vendor-only figure because the model was not listed on the public Terminal-Bench v3.0 leaderboard then. The table's steering did not come from fabricated numbers; it came from four selection choices: benchmark version, task subset, metric shape, and which rivals were included.

The audit turns a launch number into a procurement question

Keep this list near the practical procurement doc. Each item is a quick check, not a long research project.

• Who ran it? A vendor-run number is a claim about their setup; an independent runner gives you another data point. If the model is missing from a public leaderboard, ask why.
• Which harness? The benchmark harness matters because it changes the score and the cost; if the vendor does not name it, the number is under-specified.
• Is the task closed-ended? A deterministic, closed-ended test can saturate quickly and may not match your workflow.
• What is the human baseline? A baseline drawn from unselected public members is a minimum, not a maximum, and it may not cover your job.
• What is the cost and effort? A benchmark run can carry a large cost, and a model can use fewer actions than humans while still failing your task.

Use the list in order. If an answer is missing, stop and treat the number as a demo claim, not a procurement fact.

The baseline and the bill are part of the number

The ARC-AGI-3 human baseline came from about 500 members of the public, not picked for puzzle-solving skill, and humans can solve every environment in the benchmark. On action efficiency, Astra beat the human baseline on 96.0% of levels by using fewer actions, and it averaged 51.7% fewer actions per level.

ARC Prize said that saturating ARC-AGI-3 would not prove AGI and that it was not claiming GPT-6 Astra is AGI. Multiple frontier models score above 88% on MMLU, so the benchmark is saturated and less useful for deciding which model will handle a specific production workload. A saturated test can make a weak choice look safe, so check the task before you check the score.

Pricing can move independently of benchmark scores. Anthropic reduced Claude Fable 5.1's cached input price from $1.00 to $0.25 per million tokens, a 75% cut. A benchmark cost is not the same as a production cost, but it tells you whether the vendor is willing to pay for the measurement.

Keep the harness next to the score. Before you buy, run a small internal check on your own data. Pick a few tasks you already do, write the expected output, and compare the model's answer against a human answer. Keep the prompt, the model version, and the cost per run. That gives you a number that belongs to your workflow, not to a vendor's launch page.