The harness is part of the security release

An AI agent does not arrive in production as a model. It arrives inside a harness that decides which tools exist, how credentials are presented, what state survives, which actions can proceed, and what happens when something goes wrong.

That surrounding system is not packaging. It is part of the security release.

A new benchmark, HarnessRisk, makes the point unusually concrete. The authors tested the same models across different agent harnesses and found markedly different safety outcomes. Their results do not tell us which commercial deployment is safe. It gives security and platform teams a better unit of review: the deployed combination of model, harness, permissions, state, and controls.

What HarnessRisk tested

HarnessRisk is a new preprint, first submitted on August 18, 2026. It organizes agent-harness risk around six phases of operation:

• Harness Configuration: connectors, credentials, policies, and other security-sensitive settings.
• Capability Extension: installing, updating, authorizing, and managing skills or plugins.
• Runtime Operation: working with emails, webpages, documents, and tool output.
• State Persistence: writing memory, preferences, policies, identities, and triggers that can affect later work.
• Action Control: taking consequential external actions.
• Incident Recovery: investigating, containing, repairing, and preserving evidence after a suspected failure.

The benchmark contains 128 cases. Each combines a benign user goal with an adversarial instruction hidden in an untrusted workflow artifact. The agent works through three user turns with files, tools, persistent state, and controlled mock services. Every run starts from a fresh environment, real external network access is disabled, and no real credentials, accounts, payments, deployments, or other external effects are involved.

Across three harnesses, six models, and 14 model–harness configurations, the authors report attack-success rates ranging from 12.6% to 80.9%. Task utility remained between 75.0% and 97.6%. Harness Configuration had the highest mean attack-success rate on every harness in the cross-harness phase analysis.

The comparison that deserves the most attention is not the highest number. It is the movement of the same model. GLM-5.2 recorded a 54.7% attack-success rate on OpenClaw and 12.6% on Nanobot—a difference of more than fourfold.

That is not a controlled estimate of a harness-only effect. The harnesses have different prompts, tools, and state management, and the authors explicitly warn that the results compare deployed configurations rather than isolating one causal factor. That limitation is precisely why the operating lesson matters: these components arrive together in the real system, so a model-level score cannot stand in for a deployment-level review.

Configuration deserves the same scrutiny as execution

Agent-security conversations naturally gravitate toward the visible moment of action: the message sent, the file deleted, the payment initiated, or the command run. HarnessRisk found the most consistent weakness earlier, while an authorized configuration change was being made.

The dangerous value could be small: an expanded sharing scope, a public endpoint, a disabled approval step, a secret placed in the wrong field, or a reduction in redaction. The surrounding workflow is legitimate, so the unsafe parameter can hide inside work the user genuinely asked the agent to complete.

This is a familiar security pattern in a new interface. A configuration plane is a privileged workflow even when it looks like setup. It should have typed fields, safe defaults, narrow write permissions, validation at the point of change, and a reviewable diff. Natural-language intent is useful context, but it should not be the final authority for a security-sensitive value.

The same care applies to capability extension. A skill or plugin changes what the agent can reach and do. Installation, authorization, and updates therefore belong in release and change control—not in a convenience path that quietly expands the system's authority.

Recognition is not control

The study also separates detecting risk from acting safely. In two reported configurations, the agent explicitly recognized risk in more than 90% of runs while attack success remained substantial: 31.2% in one configuration and 54.7% in another.

An agent can say that content looks suspicious and still follow it. It can notice contaminated state but fail to remove the unsafe token, skill, or policy during recovery. A convincing warning in the transcript is not the same thing as a denied action or a repaired system.

That distinction should shape both product design and evaluation. Detection is an input to a control. The stronger evidence is what happened next:

• Was the unsafe parameter rejected?
• Did authorization remain bound to the trusted request?
• Was untrusted provenance preserved across later turns?
• Did the consequential action require an independent gate?
• Was contaminated state removed, and did a clean rerun confirm recovery?

These are observable outcomes. They are harder to satisfy with a reassuring explanation alone.

A release review for the whole agent system

A model change is an obvious reason to re-evaluate an agent. A harness update, connector change, new skill, permission adjustment, memory policy, network rule, or recovery procedure should trigger the same instinct.

Five practices make that review more useful.

1. Give the deployed configuration one identity

Record the model, harness, system prompt, tool and skill versions, credentials and scopes, network policy, persistent-state rules, action gates, and recovery controls as one release unit. A result should point back to that exact configuration. If any material part changes, the old result becomes historical evidence rather than a current assurance claim.

2. Test the lifecycle, not only the task

Exercise setup, extension, ordinary operation, durable state, consequential action, and recovery. A deployment can refuse an obviously malicious command while still accepting an unsafe connector setting or carrying poisoned memory into tomorrow's run. Include multi-turn cases in which the apparent trust of an artifact changes over time.

3. Measure utility and unsafe effect separately

Successful task completion does not show that the route was safe. A low attack-success rate can also hide a system that simply failed to reach the tool. Report useful completion, unsafe effects, persistence, and explicit detection as different measures. Preserve enough trajectory and state evidence to explain each result.

4. Put enforcement outside the model's narration

Use schema validation, permission checks, destination restrictions, action-specific approval, rate and cost limits, and fail-closed defaults where the consequence justifies them. The model can recommend or explain, but a separate control should decide whether the requested effect is permitted.

5. Make recovery an evaluated behavior

Test whether the system can stop, identify affected state, revoke or rotate access, remove unsafe configuration and persistence, preserve evidence, and restart from a known-good point. A run is not recovered because the agent recognized the incident. It is recovered when the unsafe influence no longer survives or acts.

What the study does not establish

HarnessRisk is a first-version preprint, not an independent reproduction or a measurement of production incident prevalence. Its results come from selected models, harnesses, cases, and sandboxed mock services. Cross-harness observability also differs, provider endpoints can drift, and some invalid runs were excluded after provider-side failures.

The benchmark is best read as a method and a warning against over-generalization, not as a product ranking. Its released cases, adapters, mock services, evaluator prompts, analysis scripts, and archived trajectory metadata should make independent follow-up possible.

The PrivacyFirst view

Agent safety belongs to the deployed system, not the model label.

The practical release boundary includes the harness that turns language into authority, the state that carries influence forward, the gates that decide whether an action may happen, and the recovery path that proves unsafe influence is gone.

A capable model inside a weak harness is still a weak system. Review the whole configuration—and follow its authority all the way through recovery.

Sources

• HarnessRisk paper and revision record
• HarnessRisk project page