Test Coding Agent Security Claims on Your Repo

Your coding agent can find a bug, patch it, and tell you the repo is safe. If it cannot show a reproducible, sandboxed, manually verified workflow on your repo, its security claims are marketing.

Mantis is an open-source framework tied to Google's internal bug-finding practice. It is not an officially supported Google product. Mantis is limited to demo use and should not be deployed in live production environments. That makes it a useful test case, not a production dependency.

The harness turns a vague promise into a measured review: find a real issue, prove it, patch it safely, and leave evidence a human can audit.

Make the test boring enough to repeat

Use the same repo, same branch, same target, and same scoring. If the agent changes the rules mid-run, the result is not comparable. A good harness makes the agent say when it cannot run a command, when a test is missing, and when a finding needs a human.

Run the five gates before you trust a fix

• Validate the preflight before the agent touches code. Confirm the repo builds, tests run, and secrets are absent. Give the agent a scoped branch, a clear file or directory target, and a test command it can run. Record the exact commands, not just the outcome. If the baseline is failing, fix it first so the agent spends its effort on the target. Stop when the baseline commit is clean, the local test command passes, and you have a short note on the agent's access scope.
• Detect capabilities instead of assuming them. Ask the agent to read, edit, run tests, and isolate changes. A useful capability check includes whether the agent can create a throwaway branch, run a targeted test, and stop when told. If it cannot run a sandbox, note that limitation before judging its output. Keep a short capability list: what it can do, what it cannot, and what you will verify by hand.
• Run a campaign against a file or directory. Do not start with the whole monorepo. Pick a module with known paths, recent changes, or a security-sensitive surface. Make the target small enough that a human can read every changed file. As the worked example, Mantis's hierarchical security-summary tree technique reduced token overhead by more than 85 percent. The gate closes with a scoped finding list, not a flood of generic warnings.
• Read the true-positive rate, not the finding count. Count only findings that survive reproduction and expert review. Sloppy AI code scanning can produce weak true-positive rates below 7 percent. If the denominator is hidden, the score is not useful. Close the gate only with a visible ratio: verified true positives divided by total findings.
• Reproduce before patching, then sandbox and manually verify. Require a reproduction script or test before the agent proposes a patch. Treat sandbox patching as a separate stage: put the patch in a disposable environment and run the relevant tests. Keep the patch and the reproduction in separate reviewable commits. Mantis findings require manual verification by a security expert before they are reported. Exit with a patch that has a reproduction, a sandbox result, and a human sign-off.

Keep the verdict honest when the evidence is messy

The common failure is treating a confident patch as proof. The agent may produce a plausible diff, a passing test, and a plain summary. None of that proves the vulnerability existed, that the patch fixes it, or that the fix does not open a new path.

In Mantis, a failed automatic reproduction of a vulnerability does not prove it is a false positive, and a successful reproduction does not guarantee it is exploitable in all contexts. The harness needs a human judgment step. The expert decides whether the finding matters, whether the reproduction is representative, and whether the patch is safe enough to ship.

Write the verdict in a brief note: what was found, what was reproduced, what was verified. If any line is empty, the agent has not finished. A patch that passes tests but cannot be tied to a specific vulnerability is a change, not a security fix.

A finding that cannot be reproduced is a lead, not a result. Use the harness to compare agents, not to replace the review. A good result is a small set of verified issues, a clean reproduction path, and a patch that survives sandbox testing. If the agent cannot produce that, mark the security-review claim as unverified.