Turn a 67% AI support claim into a defensible planning range

You are about to pilot an AI support agent, and the Intercom number in front of you is a 67% average resolution rate for Fin. AI customer support vendors publicly claim resolution rates in the 67% to 86% range. Intercom's own Fin case studies cluster from 42% to 50%. An independent 500-ticket small-business test came in at 38%. Before you sign, convert that 67% into a low/base/high range built from your ticket mix, knowledge-base quality, and failure checks.

The spread is the entry fee

Do not ask whether the vendor's headline is true. Ask what it would mean for your queue. Simple password resets may make a high resolution rate plausible. Multi-system B2B escalations make the same number optimistic: B2B AI support deployments perform 17 to 25 percentage points below vendor benchmark resolution rates.

The worksheet starts with the vendor claim and subtracts the gaps you can see. Use the low end for budget, the base end for staffing, and the high end for upside you will not promise. The range is what you can defend in a budget review; the demo number is not.

Your ticket mix sets the range

• Label recent tickets by topic, complexity, and whether a human already solved them. Stop when you have a simple table with counts, not a narrative.
• Split tickets into AI-safe and AI-risky buckets. AI-safe means the answer is documented, low-stakes, and repeatable. AI-risky means the customer needs judgment, account context, or a refund decision. You are done when paired columns can survive a skeptical manager.
• Score knowledge-base quality on coverage, freshness, and clarity. Coverage asks whether the answer exists. Freshness checks whether it matches current product behavior. Clarity tests whether a model can follow it without guessing. The output is a short memo naming the gaps, not a scorecard with no owner.
• Set the low/base/high range. Low assumes the AI handles only the easiest documented tickets. Base adds a modest share of adjacent questions to the documented slice. High tests whether the vendor claim survives your mix and knowledge-base quality. Finish with a range tied to named ticket groups.
• Apply a failure check before signing. Ask the vendor to run a sample of your real tickets. The result is a written list of false answers, unsafe actions, and escalation misses.

The common mistake is to treat a vendor demo as production. A demo can be curated; your queue is not. Even a rollout can reverse course; Klarna rehired human support staff in 2025 after earlier claiming its AI performed the work of 700 agents.

Failure checks decide whether the range survives

A pilot test should use your tickets, not the vendor's demo set. Pull a recent sample that includes the awkward cases: billing disputes, partial refunds, missing data, and customers who are already annoyed. A vendor that refuses to test on your data is a planning input, not a minor inconvenience.

Do not borrow confidence from a public sample. AI Gear Base had published 1,342 reviews after three years of operation, and that review count does not replace your own ticket test.

During the test, look for three failure modes: fluent answers that miss the issue, unauthorized actions such as promising a refund or changing a plan, and human-needed tickets kept in the AI loop long enough to create another complaint.

Write the failure list before the contract: ticket types where the AI should stop, actions it cannot take, and the handoff trigger. The end state is a short guardrail sheet that both the vendor and your support team can sign.

If the vendor's claim sits above your high end, call it a mismatch, not a problem. Tie the range to cost before you scale. A base case that only saves a small slice of human time should shrink the pilot, not expand it. Once the base case clears your cost threshold, document the assumptions and revisit after the pilot.