15-minute cache-miss test for Fable 5.1 coding agents
Run three to five real coding tasks on the old and new model, then compare success, spend, and cache behavior before repointing.
A model upgrade is worth repointing only when your own cache-miss costs and workload-specific task gains beat the current model, not when the launch table looks impressive. Run the test before you change the default model in your coding agent. Claude Fable 5.1 shipped on September 1, 2026, and the launch table made the upgrade look obvious. Use the table, but do not mistake it for your workload. The short version is small enough to run before a standup and specific enough to survive a skeptical finance review. It does not claim to prove the model is better in every setting. It only asks whether your current agent gets a measurable win on the tasks you already pay for.
The launch table is not your workload
Anthropic reported Fable 5.1 outperforming Fable 5 on Terminal-Bench 4.0, Terminal-Bench-Science 0.1, and AutomationBench. On Terminal-Bench 4.0, Fable 5.1 scored 55.8%, Claude Mythos 5.1 scored 60.9%, and Fable 5 scored 42.0%. Terminal-Bench-Science 0.1 moved from 24.7% to 52.6%, and AutomationBench moved from 17.1% to 31.4%. CursorBench 3.2.0, an IDE-style editing benchmark, moved from 70.5% to 73.4%, a smaller gap.
These are Anthropic-reported results, not independent verification that the model beats every competitor. A Financial Times review of Ramp spending data put Fable 5 at about 11% of Anthropic model spend across roughly 70,000 companies, more than two months after launch, while cheaper Opus 5 and Opus 4.8 gained share. The market had already voted with its wallet.
The short comparison is the real benchmark
Treat it as a controlled comparison, not a vibe check. You are measuring three things: whether the task succeeds, what the run costs, and whether cache behavior changes the bill. Keep the sample small. A few real tasks beat a long synthetic suite when the goal is a repointing decision.
- Pick three to five tasks from your last week of real work. Done: a short list with task name, repo, acceptance check, and the model currently used.
- Freeze the effort. Use the same prompt, same tools, same timeout, and the same acceptance check for both models. You are done when two runs differ only by model.
- Log spend by token type. Record input, output, cache-read, and any uncached context you can see. The output is a small table with model, task, success, and cost.
- Price the cache behavior. The base rates did not move: input stayed at $10 per million tokens, output at $50 per million tokens. Cache reads fell 75%, from $1.00 to $0.25 per million tokens. Finish with a bill that shows whether the cheaper cache read actually lowers your spend.
- Compare against workload analogs. If your work is terminal-heavy, weight Terminal-Bench 4.0; if it is science-lab work, weight Terminal-Bench-Science 0.1; if it is long automation, weight AutomationBench; if it is IDE-style editing, weight CursorBench 3.2.0. End with one sentence saying which benchmark analog matches your tasks.
Watch for the trap: treat a green benchmark as a green bill. A model can win the task and still lose the cost test if it burns more uncached context. If the new model needs more retries, more human edits, or a longer context window to finish the same task, that is a cost even if the final diff looks clean. Log it.
The result should read like a lab note
Repoint if the new model wins at least two of your three to five tasks and does not raise total spend. Stay put if it wins fewer tasks or raises spend, even if the launch table looks better. If it wins tasks but raises spend, check whether the cache-read cut is doing its job.
Anthropic projected typical token-billed Fable usage at roughly 25% lower cost, with highly agentic usage up to 45% lower. If your workload is not highly agentic, do not assume the larger discount applies. The restricted-access Claude Mythos 5.1 name points to the same underlying model, so treat its higher Terminal-Bench 4.0 score as a limit, not a Fable entitlement.
If you have evals, use them to tune prompts before you commit. Anthropic says teams with evals can identify a new model's strengths, adjust prompts, and complete an upgrade in days, whereas teams without evals may need weeks of testing. If you do not have evals, the short test is the minimum evidence you should require before changing the default.
Write the result as a one-line lab note: task wins, spend delta, cache behavior, and the next model to test. If the note shows more task wins and a lower bill, repoint. If it does not, leave the agent where it is.
