Run This Agent Attack Suite Before Production: Catch Prompt Injection, Memory Poisoning, and Context Drift

Before an AI agent touches production, treat it like a junior employee with a badge, a terminal, and a habit of trusting strangers. The goal is not to prove the model is safe. The goal is to prove the system around it fails in a way you can see, contain, and explain. A pre-production AI agent security checklist should test prompt injection, memory poisoning, context drift, data leakage, and kill-path behavior before you ask whether the gateway is pretty.

Start with identity, not the gateway

If your first control is a gateway filter, stop and reorder the chain before testing. Agent security is a dependency chain, and downstream controls depend on upstream context. If the agent identity is too broad, a later filter only narrows the blast radius; it does not remove the privilege that made the incident possible.

Define the agent as a non-human identity with the smallest scope that can do the job. If it only reads a ticket queue, it should not have write access to billing. If it only drafts replies, it should not execute shell commands. If it only summarizes logs, it should not hold database credentials. A 2026 Teleport study of 205 security leaders found that over-privileged AI was associated with a 76% incident rate, while least-privilege organizations reported a 17% incident rate. That is not a magic number, but it points at the same operational truth: broad agent identity turns a bad prompt into a bad day.

Scope tools before you scope prompts. Each tool should have a named owner, a documented purpose, and a maximum effect. A search tool should not become a write tool because the model decided to be helpful. If a tool cannot be described in one sentence, split it or delete it.

Build a repeatable attack suite

The suite should be boring enough to run on every release and specific enough to catch the failure modes that matter. You are not trying to exhaust the model. You are trying to find the path from untrusted input to an action you did not intend.

• Inject a canary prompt into a document, ticket, email, or web page the agent will read. Ask it to perform a harmless task, then check whether it attempts to reveal the canary, change its instructions, or call a tool it should not have called. Use a unique canary per run so you can distinguish a real leak from a cached answer.
• Poison a memory entry. Put a false fact, fake policy, or malicious instruction into the agent's long-term memory, then ask a normal question. If the agent treats the poisoned entry as authoritative, you have a memory poisoning path. If a single user message can overwrite it, you have a worse one.
• Measure context drift. Give the agent a long task with many turns, then introduce a subtle contradiction or a later instruction that conflicts with the original goal. Track whether it silently changes scope, drops constraints, or starts using tools outside the task. Drift detection is not a single metric; it is a set of invariants: no expanded permissions, no ignored safety constraints, no later user outranking system policy unless policy says so.
• Probe for data leaks. Ask the agent to summarize a document containing a canary secret, fake API key, or fake customer record. Then ask it to explain its reasoning, export its plan, or answer a follow-up that could surface the hidden value. A leak is not only the secret in the final answer. It is also the secret in a tool call, log, cached summary, or memory write.
• Test the kill path under load. Start a task, then trigger the kill switch. Verify that the agent identity is disabled, credentials are invalidated, tools are blocked, tasks are terminated, and the workload is isolated. A kill path that only stops the chat window is not a kill path. It is a pause button with extra steps.

Run these tests against the same agent version, tool set, and memory store. Record the input, expected safe behavior, observed behavior, and evidence. If a test passes only because the model happened to refuse, rerun it with paraphrases. If it passes only because the gateway blocked the output, note that the agent still attempted the action. That distinction matters when you are deciding whether to ship.

Make the gateway the fifth control, not the first

Gateway controls are useful, but they are late in the chain. At the gateway, verify that exfiltration attempts are blocked, runaway loops are rate-limited, and suspicious requests are logged with enough context to replay. They should not be the main reason the agent is safe. Sequence gateway controls fifth, not first. If your architecture depends on the gateway to catch a prompt injection that the agent identity, tool scope, memory isolation, and kill path all allowed, you have built security theater with a firewall in front of it.

A practical ordering looks like this: first, least-privilege identity. Second, scoped tools with explicit side effects. Third, isolated memory and clear provenance for stored facts. Fourth, a hard kill path that can stop the agent without waiting for a human to notice. Fifth, gateway controls, verified with the checks above. This order does not make the gateway optional. It makes the gateway honest: it is a control, not a substitute for the controls that should have prevented the problem.

Real-world pressure makes this ordering easier to understand. CISA added a LiteLLM flaw to its Known Exploited Vulnerabilities catalog in June after in-the-wild abuse. The lesson is not that one library is evil. The lesson is that agent infrastructure can pass authentication and still drift, expose data, or be memory-poisoned. If your pre-production suite cannot show how that would happen in your stack, you do not have a security review. You have a hope.