Test an MCP data backend on real agent work

You are deciding whether an MCP data backend will beat web search on your own questions. A vendor page can make the answer look settled. A model powered by Drillr MCP scored 94.06% on Vals Finance Agent. On Vals' Finance Agent v2 leaderboard, Muse Spark 1.2 leads at 60.60%, ahead of Claude Opus 5 at 58.63% and Gemini 3.5 Flash at 57.86%. Those numbers are not your workflow. That gap is the reason to test.

Under Vals' stricter All-Pass scoring, the top model fell to 50.88%, and the rest stayed under 48%. The drop is the warning: a benchmark can move a lot when the rules change. It still does not show your invoice, your filing deadline, or your messy spreadsheet.

Drillr's internal test put Sonnet 5 through Drillr MCP and through general web search on the same 200 questions, using a fixed rubric and counting each tool call. Drillr's public rows for Drillr MCP and web search used Claude Opus 4.6 on Finance Agent v1.1's public test set. Drillr designed and graded that internal benchmark, and it released the findings as company research rather than an outside audit or external certification. The run behind those Drillr figures took place on August 14, 2026. Treat those numbers as a template, not a verdict. The method is worth copying.

Run a two-arm test on your own questions

Ask whether the backend changes the answer, not whether it looks impressive. Hold the model constant and compare a structured backend against web search.

• Pick 20 real questions from your workflow. Do not let the agent choose them; if it can ask for the easy ones, the test becomes a confidence trick. Done looks like a list you would actually ask this month, not a vendor demo.
• Freeze the model, prompt, answer format, and question order. The test works when you can swap the data source without changing anything else.
• Run the same model with an MCP data backend. Look for answers that use the backend's data instead of only general knowledge.
• Run the same model with general web search. You should see browsing, but no access to your private backend.
• Score each answer against a fixed rubric before you read the next answer. A second check is that another person could score the same answer and reach the same verdict.
• Count every tool call, including retries and failed calls. That count shows whether the backend saves effort or just adds steps.
• Compare accuracy, cost, and latency across each run. End with a clear call on your questions, not on a vendor's.

Score the answer, not the demo

A good rubric should be boring. A rubric is a contract you sign before the run. It should check correctness, source citation, usability without cleanup, and made-up details. If the backend gives a clean number but the model explains it badly, the answer is still weak. If web search finds the right fact but adds extra text, the answer is still weak.

Keep the rubric fixed before you run the test. Changing it after you see the results weakens the test. Judge whether the data path changes the work, not the model's charm.

Keep the test small enough to rerun

Record the cost and latency for each answer, even if the numbers are small. A backend that is more accurate but slower or more expensive may still win for a monthly close. It may lose for a quick check. Size the test so you can rerun it after a model change, a backend change, or a shift in your questions.

When you have the results, write a short verdict: which setup was better on your questions, and by how much. If the answer is unclear, the test was too small, too vague, or too easy. Run it again with harder questions from your actual month.