AI support agent test: 300 tickets, cost per closed ticket

Start with your own queue

You are comparing AI support agents, and the vendor deck looks plausible. The only question that matters is whether the agent closes real tickets without creating extra work.

During week one, four platforms closed zero tickets without a human stepping in. The test put 12 agent platforms against one shared support ticket workload. The shared workload was built from 300 anonymized incoming support tickets at an actual B2B SaaS company.

Use that design. Take your own anonymized tickets, not vendor sample prompts, and run every candidate through the same slice of your business. The workload should include the tickets that hurt you: billing, password resets, integration errors, and the odd cases your team already knows are messy. If your queue is mostly routine, a demo about onboarding will not tell you much. If your queue is full of edge cases, the agent needs to show where it stops.

Make the test fair before you trust the score

Do not let a vendor choose the test environment. Every agent in that test got a refund endpoint capped at $50. A cap makes a refund a bounded decision, so the agent cannot hide behind an open-ended promise.

One six-platform setup loaded each system with 80 knowledge-base articles and 5,000 historical tickets, then ran 200 fresh tickets through all tools for four weeks. Copy that shape: enough history to test memory, enough new tickets to test generalization, and a long enough window to see degradation.

Build failure checks into the run. Watch for premature close: the agent marks the ticket done before the customer stops asking. Watch for memory loss: it forgets the account, the prior thread, or the refund cap. Escalation should hand a human the context, not just the ticket. The tools should match production closely enough that a good score does not depend on a cleaner test queue.

Count only closes that survive review

A close is only useful when the customer's issue is actually resolved. If the agent says closed, the metric is not trustworthy. A close-rate ladder forces the team to say what happened, not just whether the ticket changed status.

Build a close-rate ladder before you run the test:

• Fully closed: the customer's issue is resolved and no human finished the work.
• Closed with human edit: the agent did most of the work, but a person changed the answer.
• Escalated: the agent correctly stopped and handed the ticket to a human.
• Unresolved: the customer is still waiting.

In the cost results, Claude Agents ran $0.41 per closed ticket and OpenAI ran $0.38. Intercom's Fin charges $0.99 per resolved ticket, and its published case studies show resolution rates between 42% and 67%.

Run an independent cross-check. Have another person audit a sample of closed tickets against the original thread. Pick the sample randomly, not the tickets the vendor wants you to see. If the customer still has the problem, the close rate was wrong. If the agent escalated too early, the cost per closed ticket was inflated.

Calculate the final number the same way every time. Add agent fee, model cost, and human review time, then divide by the tickets that cleared the close-rate ladder. Decide before the run whether human-edited tickets count as closed. If they do, say so in the report. If they do not, the close rate drops and the cost per closed ticket rises.

Turn the run into a budget number

Keep the comparison narrow. Measure each option against the same criteria: price per closed or resolved ticket, evidence of resolution, and behavior under capped refunds. Do not let a demo replace the run.

Keep the report short enough to share with finance and long enough to defend the number. It should cover workload, tools, close definition, cost, and the failures you saw. Treat published pricing and test figures as references, not a ranking.

Use your own run's cost per closed ticket for the next budget conversation. If the queue changes, rerun the test before you change the vendor.