On May 25, 2026, Anthropic published a post describing how it contained Claude in the face of prompt injection attacks Anthropic, “How We Contain Claude”. The post included specific numbers: 93% of permission prompts approved by users, auto mode catching approximately 83% of overeager behaviours before execution, approximately 0.4% of benign commands blocked, approximately 17% of overeager actions getting through, Claude Opus 4.7 prompt-injection resistance at approximately 0.1% on single attempts and 5–6% after 100 adaptive attempts. The post also stated a design principle: contain the agent, because you cannot put a ceiling on what gets through.
This article reviews the published benchmark evidence behind that principle. Each number below comes from a paper with an open evaluation methodology. I link to the primary source where each number appears, note the measurement conditions, and explain what the number actually means. The goal is not to argue for or against containment, but to present the numbers as they were measured and what those measurements support.
Static benchmark results
These results come from fixed-benchmark evaluations, where a set of attacks is defined in advance and each defence is tested against them.
CaMeL Debenedetti et al., Google Research, arXiv 2503.18813 wraps the model in a layer that reads the plan from the trusted request first, then runs it. Data the agent retrieves can feed values into that plan but cannot change its structure. Tool calls are checked against policy before execution.
On AgentDojo Shumailov et al., arXiv 2410.02644, a benchmark with 97 user tasks and 629 security cases, CaMeL completes 77% of tasks with a security guarantee that holds by construction. An undefended system completes 84%. The seven-point gap is capability loss, not security failures. Within CaMeL’s assumptions, the remaining failures are tasks the model could not finish, not attacks that succeeded. This means CaMeL’s security guarantee is structural — it does not rely on the model behaving correctly, which is why the gap is so small.
Progent Narisetty et al., arXiv 2504.11703 is a privilege-control framework with its own policy language. On indirect injection, it reduces attack success from approximately 39.9% to about 1.0%. On the Agent Security Bench Shumailov et al., arXiv 2410.02644 — a benchmark covering 16 attack types, 11 defences, 10 scenarios, and 400+ tools — it reduces attack success from 70.3% to 3.9%. The large reduction on the Agent Security Bench is notable because that benchmark tests a wider range of attack types than most.
VIGIL Lin et al., arXiv 2601.05755 adds a verify-before-commit step ahead of execution. It reduces attack success by more than 22% compared with the best existing dynamic defences. This is an improvement on an already-defended baseline, not a comparison to an undefended system — meaning the 22% represents the additional margin VIGIL provides on top of whatever defence was already in place.
MCP-SafetyBench Wu et al., arXiv 2512.15163 reports host-side attacks — intent injection and identity spoofing — succeeding more than 80% of the time on average across 20 attack types. This is the undefended starting point. The high rate reflects that host-side attacks target the infrastructure around the model, not the model itself, and most evaluations do not include host-side defences.
Grouped bars show undefended vs defended attack success rates across benchmarks. The bottom row shows dynamic environment degradation from AgentDyn.Measurement limitations
The numbers above are real. They come from published papers with open evaluation code. Each one depends on conditions that limit how far the results generalise.
Fixed benchmarks measure familiarity, not robustness
Newer frontier models score near-zero attack success on AgentDojo and InjecAgent with no defence at all. This could reflect model improvement, or it could reflect benchmark saturation.
AgentDojo uses the same 629 security cases across evaluations. InjecAgent uses the same 1,054 indirect-injection cases. Models learn to resist the specific patterns in these benchmarks. Nobody is measuring what happens against an injection the model has not seen.
A fixed set of cases is necessary for reproducibility. Reading a maxed-out score as general robustness is the limitation.
The adaptive attack result
LaunchSafe’s adaptive evaluation Narisetty et al., arXiv 2606.26479, June 25, 2026 runs a different experiment: it takes a static benchmark, then rewrites the attacks to target the specific defence in front of the evaluator. This is the experiment the field needs — a defence that only works against attacks it has seen is not a defence, it is a filter for a known list.
Against Progent, the undefended attack success was 25.8%. The standard attack against Progent got 4.2%. The adaptive attack got 2.6%. The paper’s Finding 2 states: “Mean ASR under our adaptive attack was 2.6%, lower than the standard attack against the same defense (4.2%), not higher.”
The authors note this is “one small-scale data point on a weak model with a single black-box attack template,” and that attacks with white-box access to the model remain untested. The result does not show that adaptive attacks always perform worse — it shows that the relationship between standard and adaptive attack performance depends on the defence and the model, which means no single number captures the full picture.
Dynamic environments
AgentDyn Li et al., arXiv 2602.03117 tests defences in dynamic environments where the agent receives new content, new tools, and a new context window during execution. The paper’s conclusion: “almost all existing defenses are either not secure enough or suffer from significant over-defense.” The over-defense finding is important: a defence that blocks so much that the agent becomes useless is still a failure, just in the opposite direction. This is what happens when a defence is tuned to catch attacks but not calibrated to preserve utility.
The over-defense finding — a defence blocking so much that the agent becomes useless — is often omitted from summaries. In AgentDyn’s numbers: CaMeL’s utility drops to 0% once the environment moves, Meta’s SecAlign falls from 80% to 53.35%, and PromptGuard2 still lets through 27.15% of attacks.
RAS-Eval (Li et al., arXiv 2506.15253), a benchmark built around real execution rather than fixed replay, finds a similar pattern: failure modes double under live conditions (32 vs 16) and average task completion drops by 36.8%.
The difference between benchmark and production is structural. A benchmark tests against a fixed set of attacks. Production presents new content, new tools, and new context continuously. A defence that works in one setting may block legitimate work in the other.
Survey statistics
A separate class of numbers in this space comes from incident-rate surveys. These report widely varying figures for claims like “X% of deployed agents are vulnerable to prompt injection.”
The variation is methodological. Each survey sampled a different population, defined “vulnerable” differently, and none published a measurement method. Without knowing the denominator or the definition, these numbers cannot be compared or combined.
A taxonomy paper Li, Fung et al., arXiv 2605.16282 documented this inconsistency across published agent-safety benchmarks: definitions of “success,” “attack,” and “vulnerable” vary enough between papers that head-to-head comparison requires careful alignment. This is not just a semantic problem — it means that a paper reporting “90% vulnerability” and another reporting “30% vulnerability” may be measuring fundamentally different things.
One number that does include its methodology is the AppSec Santa audit AppSec Santa, April 2026, which found a 78% false-positive rate in pattern-matching MCP scanners. The working: 33 servers scanned, 433 tools tested, 27 detections, 6 of them real. This number can be evaluated because the method is stated — 33 servers scanned, 433 tools tested, 27 detections, 6 of them real. The 78% false-positive rate means that for every real detection, there were roughly three false alarms, which has practical implications for any system that relies on pattern matching to block tool calls.
What the evidence supports
The published results support three conclusions:
Static benchmarks show real defensive capability. Defences like CaMeL, Progent, and VIGIL reduce attack success by large margins against the attacks they were evaluated on.
Those margins depend on measurement conditions. Fixed benchmarks saturate. Adaptive attack performance varies by defence and model. Dynamic environments degrade utility. None of these invalidate the static results; they define the conditions under which those results hold.
Containment addresses a gap that defences alone do not close. A defence reduces attack success against a known set of attacks. Containment limits damage when an attack succeeds or when an attack exists that has not been measured. The Anthropic containment post frames this as a design principle rather than a claim about defence performance.
Design implications
The measurements point toward containment as a complementary control, not a replacement for defence layers.
Isolating the environment — keeping the model’s context separate from the data it processes, limiting which tools it can reach, capping how much data a tool call can return — is the control that survives adaptation. It is also the hardest to implement well.
The joint CISA guidance suggests an incremental approach: isolate the environment first, add verification steps, cap data returned by tool calls, and accept that residual risk never reaches zero.
Benchmark summary
| Benchmark | Coverage | Undefended ASR | Defended ASR | Notes |
|---|
| AgentDojo | 97 user tasks, 629 security cases | ~25% | 0% (frontier models) | Fixed cases; saturation possible |
| Agent Security Bench | 16 attack types, 11 defences, 10 scenarios, 400+ tools | ~84% | 3.9% (Progent) | Highest average attack success across benchmarks |
| InjecAgent | 1,054 indirect-injection cases | ~24% | 0% (frontier models) | Fixed cases; indirect injection only |
| WASP | End-to-end web agent settings | Up to 86% partial success | — | Authors describe as “security by incompetence” |
| MCP-SafetyBench | 20 attack types across server, host, user | >80% host-side | — | Host-side attacks (intent injection, identity spoofing) |
| RAS-Eval | 80 test cases, 3,802 attack tasks, real execution | 16 failure modes | 32 failure modes under live conditions | Real execution; task completion drops 36.8% |
Primary sources
- CaMeL, Defeating Prompt Injections by Design (Debenedetti et al., arXiv 2503.18813): https://arxiv.org/abs/2503.18813
- Design Patterns for Securing LLM Agents against Prompt Injections (Beurer-Kellner et al., ETH Zurich / Invariant Labs, arXiv 2506.08837): https://arxiv.org/abs/2506.08837
- Progent: Securing AI Agents with Privilege Control (arXiv 2504.11703): https://arxiv.org/abs/2504.11703
- VIGIL (Lin et al., arXiv 2601.05755): https://arxiv.org/abs/2601.05755
- Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents (Narisetty et al., LaunchSafe, arXiv 2606.26479): https://arxiv.org/abs/2606.26479
- AgentDyn: Are Your Agent Security Defenses Deployable in Real-World Dynamic Environments? (Li et al., arXiv 2602.03117): https://arxiv.org/abs/2602.03117
- Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents (Li, Fung et al., arXiv 2605.16282): https://arxiv.org/abs/2605.16282
- A Survey on Agentic Security: Applications, Threats and Defenses (Shahriar et al., arXiv 2510.06445): https://arxiv.org/abs/2510.06445
- Agent Security Bench (arXiv 2410.02644): https://arxiv.org/abs/2410.02644
- InjecAgent (arXiv 2403.02691): https://arxiv.org/abs/2403.02691
- WASP (arXiv 2504.18575): https://arxiv.org/abs/2504.18575
- MCP-SafetyBench (arXiv 2512.15163): https://arxiv.org/abs/2512.15163
- RAS-Eval (arXiv 2506.15253): https://arxiv.org/abs/2506.15253
- Anthropic, How We Contain Claude: https://www.anthropic.com/engineering/how-we-contain-claude
- AppSec Santa, YARA-based MCP scanner audit (April 2026): https://appsecsanta.com/research/mcp-server-security-audit-2026
- Five Eyes guidance, via Cloud Security Alliance summary: https://labs.cloudsecurityalliance.org/research/csa-research-note-cisa-agentic-ai-guide-enterprise-implement/
Earlier in this series
