Test an AI gateway on real agent traffic: five probes

A gateway looks impressive in a slide deck until an agent loop starts calling it. Gartner forecasts that 70% of engineering teams that build multimodel applications will depend on AI gateways for reliability and cost control by 2028. The forecast is not a test result. For a small multi-model stack, the question is whether the gateway can keep routing, guardrails, logs, and cost records predictable before it becomes the default front door for agents.

Run five probes before you trust the dashboard

Each probe should be quick enough to run in staging and repeatable. You do not need production traffic, but you do need a realistic agent path: a user prompt, a tool call, a model response, and a cost record. A vendor demo hiding the raw event has not proven the feature.

• Routing: Send the same task to a cheap model and an expensive model, then confirm the gateway chooses the intended one. A routing rule should be visible in the request metadata, not buried in a config file. Static endpoint routing alone will not survive a stack where cheap models handle extraction and expensive models handle reasoning.
• Guardrails: Submit a prompt that asks the model to reveal system instructions, then submit another prompt containing a fake internal identifier. The gateway must block or redact the unsafe part before the model sees it and inspect the response for leaked data. A pass means the block is visible in the event stream, not just a generic error.
• Observability: Trigger a failed request, a blocked request, and a successful request. The dashboard should show the model, the latency, the token count, the policy outcome, and the caller. Observability is decorative when you cannot trace an agent turn from prompt to response.
• Policy enforcement: Create an allowed identity and a denied identity, then run the same request through both. Access enforcement belongs at the request boundary: record the denial and expose the policy decision in an audit log. A model provider key as the only control makes the gateway mostly a proxy.
• Cost attribution: Run requests from different teams or workloads, then open the billing view. Spend must be assigned to the responsible caller, not just to a shared API key. A monthly total visible to finance means the gateway has not solved the problem.

Vendor claims set the bar, but they are not proof

Snowflake introduced dynamic routing in Cortex AI Gateway so companies can pick from several AI models depending on the task. Ask every vendor to demonstrate that capability with a live request, not a screenshot. Snowflake says Cortex AI Gateway can work with more than 100 MCP servers and provide a single place for access policy, identity checks, permissions, and audit records. A competing product that cannot show the same controls in one place is selling a bundle of point tools.

F5 describes its AI Guardrails as checking each prompt and response, removing sensitive data before model calls, stopping injection or jailbreak attempts, and refusing requests that cannot be evaluated. Use that as the minimum for a guardrail probe: fail closed rather than quietly pass traffic when the guardrail cannot evaluate the request.

Cost attribution is the real test

F5 claims its AI Gateway can lower token spend by up to 60% through real-time attribution and proactive budget enforcement, without changing applications. AT&T has described a proprietary AI gateway that it says is designed to reduce AI costs by up to 90%. Those numbers are useful only if the vendor can show the mechanism: a per-request cost label, a budget limit, and a denial or reroute when the limit is reached.

Snowflake says Cortex AI Gateway gives IT and finance teams a consolidated view of AI usage, ties spend to the responsible team, agent, or workload, and applies spending limits before costs grow. Ask for the same view in your own staging tenant. Cost records that appear only in a vendor console are not auditable by your finance team.