Hidden reasoning is not a security boundary

An interface can hide a model's reasoning without making that reasoning confidential.

That distinction matters as reasoning models take longer paths, call tools, and carry working state across several steps. The user may see only a short answer or a compact summary. The surrounding system may still preserve much more so the model can continue its work.

A new preprint called EchoCoT examines whether that preserved state can be drawn back out. Its results are early and need independent reproduction, but the control lesson is already useful: if information remains available to a running system, "not displayed" is not the same security property as "not reachable."

The replay surface between tool calls

Ordinary multi-turn conversations commonly discard hidden reasoning after a turn. Tool use creates a different flow. A model may need to preserve its reasoning while it waits for a function result, then continue from that state when the result returns.

The EchoCoT researchers call this a reasoning replay surface. Their attack places adversarial feedback in a tool result and repeatedly asks the model to expose more of the reasoning it retained within the same turn. Signals returned by the API, including reasoning-token counts and sometimes a compressed reasoning summary, help the attack estimate whether an extracted candidate is becoming more faithful.

The paper evaluates three open-source reasoning models and five proprietary models. For the open models, the researchers had access to the target traces and could compare the extracted text directly. Under their more practical success criterion, which allows a 10% length difference and requires at least 90% exact-token overlap, the reported success rate reached 66.4% on the in-domain test set and as high as 80% on selected unseen datasets.

Those are the paper's results, not a universal rate. They come from selected models, datasets, tool flows, prompts, and API behavior.

The proprietary-model results require even more care. The researchers could not see the ground-truth hidden reasoning or use the providers' original tokenizers. They compared extracted material with reported reasoning lengths and available summaries instead. Close length and semantic agreement are evidence worth investigating, but they do not prove near-verbatim recovery with the same certainty as the open-model experiments.

The study also used a single run per sample, so it does not measure run-to-run variation. The authors say they disclosed the findings to Google, Anthropic, and OpenAI and withheld provider-specific optimized attack trajectories from public release.

Tool output belongs on the untrusted side

The vulnerable path is not simply "the model thought too much." It is a system interaction:

• hidden state remains available so a task can continue;
• an external tool is allowed to return new content into that state;
• the model can place material in observable tool arguments; and
• interface signals help an attacker judge whether repeated attempts are working.

Reviewing each component separately can miss the loop. A tool may be properly authenticated and still return hostile text. A reasoning trace may be hidden from the chat interface and still influence later tool calls. A token count may look like harmless billing metadata and still provide useful feedback to an adaptive attack.

This is why tool results should keep their untrusted provenance even after they enter a model's context. The system should not treat a value as trusted merely because an authorized connector returned it.

Removing one signal is not the same as closing the path

The paper tests several defenses. Removing reasoning state after a tool call stopped the tested replay attack, but the authors describe that as an idealized upper bound: the model may need the state to complete legitimate work.

A defensive system instruction reduced extraction substantially in the tested open models, although an adaptive attacker recovered some success. Removing a reported reasoning-token count also helped, but total usage could still provide an estimate. Removing all length information reduced the feedback available to the attack, yet did not remove the underlying reasoning continuity.

The details will change as providers update their systems. The durable conclusion is that no single presentational change should carry the whole security claim.

Treat hidden state as a protected asset

Teams building or buying reasoning agents can begin with five practical steps:

1. Keep secrets out of reasoning. Do not place credentials, recovery material, privileged policy text, or unnecessary personal data in hidden working state simply because users cannot see it.
2. Map continuity across tools. Document what state survives a tool call, who can supply tool results, what the model can emit in arguments, and when the state is finally discarded.
3. Preserve provenance. Carry the fact that tool output is externally supplied into later policy and authorization decisions.
4. Test the sequence. Red-team multi-step tool interactions, retries, and adaptive feedback—not only one prompt followed by one answer.
5. Layer the controls. Minimize exposed fidelity signals, constrain tool schemas and outputs, add model-level resistance, monitor repeated extraction patterns, and keep consequential authorization outside the model.

These measures will not prove that hidden reasoning can never leak. They make the boundary clearer and reduce the value of any one failure.

The boundary follows reachability

EchoCoT is a first-version preprint, not a production incident report or a final measure of proprietary-model exposure. Provider updates may narrow or close the tested paths. Independent reproduction may strengthen or weaken the findings.

That uncertainty should narrow the claim without erasing the operating lesson.

Our view is simple: protect information according to where it can flow, not where the interface chooses to display it. If a model can still recall state, a tool can still influence it, and an output can still reveal it, that state belongs inside the security boundary.

Hidden is a useful interface property. Confidential requires more.

Sources

• EchoCoT paper and revision record