Untrusted input

Why a model cannot reliably tell data from instructions, from the published research; and what that does to every decision a classic agent makes after it reads something untrusted, against a plan fixed in advance.

Loading the animation…

Concept

A language model reads one stream of tokens. The user's request, the system's instructions, a web page a tool fetched and a message from a stranger all arrive in the same stream, and nothing in the model's architecture marks which parts are instructions to follow and which are data to work on. Untrusted text in a tool result can therefore steer the model: change which tool it calls next, with which arguments, and where the result goes. This is indirect prompt injection, named and catalogued by Greshake et al. (2023) after Perez and Ribeiro (2022) showed direct goal hijacking; Liu et al. (2023) formalise it and benchmark defences against it.

It is measured, not hypothetical. InjecAgent (Zhan et al. 2024) built 1,054 test cases across tool-using agents; a ReAct-prompted GPT-4 agent was steered in 24% of them, and reinforcing the injected text nearly doubled that. AgentDojo (Debenedetti et al. 2024) has 97 realistic tasks and 629 security test cases, and its authors found it a challenge for attacks and defences alike: the attacks they ran broke some security properties but not all. Prompting defences help: marking which text came from where (Spotlighting, Hines et al. 2024) cut attack success from > 50% to < 2% in its experiments. But a lower rate is not zero (chapter 6 does the arithmetic), so this site assumes the worst: once a model has read untrusted text, anything it decides next may be the untrusted text's decision.

The animation shows what that assumption does. In a classic agent the model reads every result before choosing the next call, so the control label of each call is the join of everything returned so far. In the task "send a colleague a summary of my notes, after looking up a page", the page lookup is already private-controlled (the model has read the notes, so the address it requests could carry them) and goes to the user for approval; after the page comes back, every later choice is untrusted and private, and the final send is deny. Planned in advance, the calls are fixed from the request alone and only the plan's arguments carry labels: the summary is made from the notes only, so the send carries trusted private data and goes for approval (ask), and the page lookup is just allowed. Same task, same policy; what changed is what the decisions depend on.

Maths

Number the calls $1 \ldots n$ and let $r_j$ be the result of call $j$ . In a classic agent the model chooses call $k$ after reading $r_1 \ldots r_{k-1}$ , so its control label is

c_k = \ell(\text{request}) \sqcup \bigsqcup_{j<k} \ell(r_j),

which can only rise with $k$ : one untrusted result makes every later control label untrusted. Planned in advance, $c_k = \ell(\text{request})$ for every $k$ , and a call's effective label is the join over its explicit arguments only. The engine's tests compute every call's label a second way, by tracing each value back to the reads it came from (its provenance), on every plan of up to three calls, and require the two to agree.

Plan-then-execute does not make the model trustworthy: a value derived from untrusted data is still untrusted, and the policy still applies to it. What it removes is untrusted control: no text the agent reads can add a call, remove one, or change where a result goes. Beurer-Kellner et al. (2025) list it among six design patterns with this property; chapter 7 returns to them.

Code

The line that separates the two agents, cut from the vendored engine (policy/engine.ts): the control label is the whole context, or, planned in advance, the join of the values the step branches on (ctrl, none in these tasks, so the bottom of the lattice; chapter 7 uses it).

const control: L.Label =
  mode === "context" ? { ...ctx } : L.joinAll(ctrl.filter((a) => a in vals).map((a) => vals[a]!));