Test AI Agent Memory on 100 Support Conversations

Memory failure is the rollout risk

A customer says the refund was promised. The agent asks for the order number again. That repeated ask is the failure mode to test before an AI support agent meets customers. It is a context-management problem that can be measured.

Agents already stumble on multi-turn enterprise work. In the Salesforce CRMArena-Pro (2025) benchmark, multi-turn success fell to 35% from 58% on single-turn work, and context loss led the failure modes. If a model can drop a task, a support agent can drop a person. The fix is a repeatable evaluation, not a longer prompt.

Build the harness around support context

Treat memory as structured context the agent can retrieve, update, and discard. Production agents at companies including Intercom, Notion, and Replit implement multi-tier memory. Support context has layers: account facts, open issues, promises, and policy constraints. A transcript dump can hold all of it, but it does not tell the agent which layer is current, which layer is private, or which layer should trigger the next action.

Memory design changes accuracy, tool use, and latency. Agents grounded in governance metadata improved SQL accuracy by 38%. In Atlan AI Labs' 2026 internal benchmark, adding an ontology layer improved agent answer accuracy by 20% and reduced tool calls by 39%. In Mem0's April 2026 arXiv paper 2504.19413, memory-augmented agents achieved 91% lower p95 latency and 26% better performance than default memory approaches.

The common mistake is testing with fresh prompts and calling the result memory. A fresh prompt proves single-turn answering. It does not prove the agent kept the customer's state across sessions. The harness must separate what the agent saw, what it stored, and what it can retrieve later.

The test should run where the agent will fail

• Collect a fixed set of 100 multi-session support conversations. Keep the set stable, with customer IDs, timestamps, and expected facts, so a score means something across runs. The conversations should include follow-ups, corrections, and policy questions, not only clean first-contact tickets.
• Label the facts the agent should remember: account state, open issues, promises, and policy constraints. Each fact needs a source turn and the next action it affects. If a fact does not change what the agent should do next, it is not a memory test fact.
• Strip the live context and replay only the memory layer. The agent should answer from stored context, not from the full transcript. Hide the original conversation from the prompt, then ask for the customer's current state and the next step.
• Score recall, stale facts, and leaks. Recall is whether the agent retrieves the right fact. Stale facts are old values presented as current. Leaks are private details from one customer appearing in another. Output a pass or fail for each fact, with the source turn and the affected action.
• Compare the result against a pass line and a cost ceiling. Fail the agent if it misses a fact that changes the next action. At GPT-4o input pricing of $2.50 per 1M tokens, a heavy user generating 100 50-turn conversations per day, each exceeding 20,000 tokens, would pay $5 per day in context alone.

Make the rubric hard to game

Score each fact separately, not each conversation. A conversation can look fine while hiding a missed promise. Force the agent to state its remembered facts, its unknowns, and its next step. If it guesses, mark it as a stale fact or a leak, depending on what it got wrong. A confident wrong answer is the failure a customer will feel.

A leak is a hard fail even if the answer is otherwise useful. Rerun the full set when the model, memory store, retrieval policy, or prompt changes. The harness tests retrieval policy, because memory is useful only when it surfaces at the right moment. Keep the conversations, labels, and scoring rules boring, so the only variable is the agent you are trying to ship.