Give the attacker a second move

A defense can perform beautifully against yesterday's attacks and still be poorly prepared for tomorrow's attacker.

That is the central challenge in research presented at USENIX Security 2026. The authors examined 12 recent defenses against language-model jailbreaks and prompt injection. Many had reported attack-success rates near zero under their original evaluations. When the researchers let attacks adapt to the defense, most were bypassed at rates above 90%.

The result is not a reason to abandon filters, model training, prompt structure, or other safeguards. It is a reason to ask a better question of every security score: what kind of attacker was allowed to try?

A benchmark and an adversary are not the same thing

A static benchmark presents a fixed collection of known attacks and records how a system responds. It is useful for repeatability, regression testing, and comparison. It can show that a defense blocks the cases in that collection.

But a real attacker does not keep submitting the same failed string. They observe what happened, change the wording or strategy, learn which parts of the system matter, and try again. They may know how the defense works. They may optimize against its confidence score, exploit the evaluator's success criterion, or find a plausible instruction that is difficult to classify as malicious without rejecting benign work too.

The USENIX paper calls this an adaptive attacker. Its general loop is simple: propose candidate attacks, score them against the defended system, keep the most promising, and update the next attempt. The researchers instantiated that loop with gradient methods, reinforcement learning, search, and human red teaming.

Across the defenses they tested—including prompt-based protections, adversarial training, standalone detectors, and mechanisms that rely on hidden information—the adaptive attacks found routes around the intended protection. In a selected set of 29 agent scenarios, human red teamers collectively found a successful attack in every scenario. Automated search was easier to repeat at scale, while people were particularly effective at semantic and context-dependent strategies.

This does not establish that every deployment will fail at the same rate. The research evaluates different defenses under their relevant models, benchmarks, access assumptions, success criteria, and attack budgets. Those figures should not be flattened into a league table. The underlying preprint also dates to October 2025; its peer-reviewed USENIX publication is the current milestone, not the first appearance of the finding.

What a low attack-success rate actually means

An attack-success rate is always conditional. It describes what one evaluator observed for a particular system, under a particular attacker and budget, at a particular time.

If the attacker used a static corpus, the result says how the system handled that corpus. If the attacker could query the defense repeatedly and optimize from the feedback, the result supports a stronger claim. If expert humans knew the design and still failed within a meaningful budget, it is stronger again.

None of these results proves that no successful attack exists. Empirical security evaluation can expose a break; it cannot prove the absence of every break. Its value lies in making a robustness claim harder to overstate and easier to test again.

This distinction matters in procurement and release reviews. A dashboard may show “2% attack success,” but that number is incomplete without the model and application versions, defended surface, attacker knowledge, number of attempts, compute and human effort, success definition, and unresolved failure modes.

A practical adaptive-evaluation brief

Before accepting a prompt-injection or jailbreak result, ask for evidence in seven areas.

The most useful program has two connected tracks. The first is a stable regression suite: known attacks, benign lookalikes, and important business journeys that run often enough to catch an obvious step backward. The second is an adaptive exercise in which the evaluator can study failures, change strategy, and spend more effort on the paths that appear promising.

Keeping those tracks separate prevents a common reporting mistake. A regression suite answers, “Did we reintroduce a known weakness?” An adaptive exercise asks, “What can a motivated attacker discover now?” A defense may improve substantially on the first measure while remaining fragile on the second. Both results matter, but they support different claims.

The exercise should also include ordinary work. A defense that blocks every attack by refusing useful requests has not solved the problem; it has moved it into availability and adoption. Measure whether legitimate users can still complete the same journeys, especially those with ambiguous documents, external content, or high-value tool calls. Security and utility belong in the same report.

1. Name the exact system under test

Record the model, application prompt, retrieval and memory paths, tools, permission configuration, detectors, downstream authorization, and environment. Testing a model response is not the same as testing an agent that can read mail and send a message.

2. Describe what the attacker knew

Did the evaluator know the defense design, hidden prompts, decision thresholds, or tool schema? Could it observe confidence scores, only final responses, or the real external effect? Security that depends on obscurity should be described that way.

3. Give the attacker room to adapt

Allow repeated attempts that use prior outcomes to improve the next proposal. Mix automated search with human red teaming. Static suites remain useful, but they should be the regression floor, not the ceiling of the evaluation.

4. State the budget

Record queries, wall time, compute, model cost, and human effort. A defense that survives ten attempts and one that survives a well-resourced campaign have not established the same thing. When resources are limited, a smaller number of deeply tested high-impact scenarios can be more informative than a broad run of weak attacks.

5. Measure the real outcome

Define success at the application boundary. Did the model merely emit suspicious text, propose a tool call, pass downstream authorization, disclose protected data, or change an external system? Automated judges are useful at scale, but they are also models and can be manipulated or simply mis-score the result. Validate consequential successes and failures against harder evidence.

6. Report the lower bound honestly

List the attacks that succeeded, the controls they crossed, the cases that remain untested, and the strongest attacker that failed. Avoid turning “we did not find a break” into “the system is secure.” A detector may still block many opportunistic attacks even when it does not withstand a determined adaptive one.

7. Make re-evaluation a release condition

Models, system prompts, tools, permissions, retrieval sources, and policies change. Each material change can invalidate an earlier result. Preserve the evaluation configuration and rerun both static regressions and adaptive tests before expanding authority or making a stronger claim.

The PrivacyFirst view

Robustness is not a score by itself. It is a claim bounded by an attacker, a budget, a configuration, and a moment in time.

Good evaluation makes all four visible. It lets static tests catch regressions, automated attacks explore at scale, and skilled people challenge the assumptions the automation cannot see. Most importantly, it keeps a passing result in its proper place: useful evidence from a serious attempt to find a failure, not a certificate that no failure remains.

If the attacker never gets a second move, the evaluation may be testing memory. Give them the chance to learn, adapt, and try again. That is when a security claim starts becoming useful.

Sources

• USENIX Security 2026 paper page
• USENIX Security 2026 full paper
• arXiv preprint and revision history