Score Agent Memory on Recall, Latency, Cost, and Leakage Before It Owns Your Support Queue

Memory is the support queue's hidden failure mode

Agent memory sounds like a feature. In a support workflow, it is a liability until proven otherwise. A customer asks about a refund, a failed login, and a billing dispute across three channels. The agent needs the right facts from turn one, the right policy from turn four, and the wrong nothing from a different customer's case. If the memory layer is slow, expensive, or leaky, the queue does not just get slower; it gets unsafe.

The market is crowded enough that a casual vendor demo is not enough. A 2026 ranked list of 20 AI agent memory and context tools for production gives you a starting point, but the median agent readiness score across those 20 companies is 51 out of 100. That is a useful warning: the category is not uniformly production-ready. Redis earns the highest agent readiness grade among the tools reviewed. None of that tells you whether a tool will keep your support context intact under real ticket volume.

The practical move is to stop asking 'which memory tool is best?' and start asking 'which tool survives a 30-day support workflow?' You need a bake-off that measures recall, latency, cost, and leakage in the same order your support org feels them: first, did the agent know what it needed? second, did it know it fast enough? third, did it cost less than the ticket it saved? fourth, did it keep one customer's data out of another customer's conversation?

Define 50 tickets before you touch a tool

Your evaluation set should look like your queue, not like a benchmark. Build 50 multi-turn support tickets from anonymized production data, or from a realistic synthetic mix if privacy rules block raw tickets. Each ticket should have a known answer, a known set of required facts, and a known set of forbidden facts.

1. Use 10 tickets that are short and factual: order status, password reset, shipping delay. These test basic recall and low-latency retrieval.
2. Use 10 tickets that are multi-turn and procedural: a refund request that changes from partial to full, then requires manager approval. These test context management across turns.
3. Use 10 tickets that are long-context: a customer pastes a long error log, a policy excerpt, and a prior chat transcript. These test whether the tool can find the relevant needle without dumping the whole haystack into the prompt.
4. Use 10 tickets that are cross-channel: email, chat, and phone notes. These test whether memory can bind the same customer identity across formats.
5. Use 10 tickets that are adversarial or ambiguous: a customer asks about another account, a support agent asks for a policy that does not exist, or a ticket contains a prompt-injection-style instruction. These test leakage and refusal behavior.

For each ticket, write a rubric before the test. A good rubric has three parts: required facts, forbidden facts, and acceptable answer. Required facts are the minimum the agent must retrieve. Forbidden facts are the data it must not retrieve, such as another customer's email, a different account's billing ID, or a policy that applies to a different region. Acceptable answer means the agent can be right in more than one phrasing, but it cannot be right by guessing.

Run the same 50 tickets through each tool under the same model, prompt template, and retrieval settings. If you are comparing RAG pipelines, keep chunking constant. If you are comparing long-context windows, do not let one tool get a larger context budget than another. If you are comparing managed services, record the exact plan, region, and concurrency limit. The goal is not the most impressive demo. The goal is the tool least likely to embarrass your support team on day 31.

Score the four axes, then cut tools that fail two

Score each tool on four axes. Keep the scoring simple enough that a support-ops engineer can defend it in a postmortem.

• Recall accuracy: percentage of required facts retrieved and used correctly. A tool that retrieves the right fact but buries it under irrelevant context is not fully correct. A tool that answers from a stale summary is not correct. Score 100 percent only when the final answer contains every required fact and no material error.
• p95 latency: the 95th percentile time from ticket update to agent response. Support queues are not batch jobs. If p95 latency is 8 seconds for a simple order-status question, the agent feels broken even if recall is high. Set a channel threshold before the test: chat may need sub-3-second p95, email can tolerate more.
• Cost per 1,000 resolved tickets: include storage, retrieval, embedding, model tokens, and any managed-service fees. Do not score only the memory database. A cheap vector store with expensive reranking and long prompts can lose to a pricier managed service that uses fewer tokens. Express the number in US dollars per 1,000 resolved tickets, and include a sensitivity case for a 20 percent ticket-volume spike.
• Leakage incidents: count any case where the tool retrieves or exposes forbidden facts, mixes customer identities, or lets one ticket's instructions affect another ticket. This is the axis with the highest penalty. A single serious leakage incident should be treated as a production blocker, not a minor deduction.

Use a pass/fail threshold for each axis. For example: recall at least 90 percent, p95 latency under your channel target, cost under your unit-economics ceiling, and zero critical leakage incidents. Then apply the cut rule: any tool that fails two axes is out. This prevents a tool with brilliant recall but unacceptable leakage from winning on one strong axis. It also prevents a cheap, fast tool from hiding a recall problem behind a low price.

When you write the results, publish the failure pattern. 'Tool A failed leakage and latency' is more useful than 'Tool A scored 82.' It tells the next team what to watch for. If a vendor appears in your shortlist, treat it as a candidate, not a conclusion. Your queue is the final reviewer.