Output-handling / downstream-injection testing

How it works

When model output is passed unsanitized into a shell, a SQL query, a browser, a rendering pipeline, or another LLM, the vulnerability class is a familiar one — injection, SSRF, XSS, remote code execution — just arrived at by a new route. Testing this means exercising the actual downstream sink with adversarial completions, whether elicited through prompt injection or written directly as red-team input, and asserting invariants at the boundary: every output escaped, validated against a schema, or checked against an allowlist before it's used, exactly as if it came from an anonymous user. This applies equally to natural-language output and to code the model writes and something else executes — the model's fluency is not a safety property of the sink.

When to use it

Any system that renders, executes, queries, or otherwise acts on model output without a human reviewing it first — code execution, database queries, markdown/HTML rendering, shell commands, calls to other services; and again whenever a new downstream sink is added, since coverage on one sink says nothing about the next one.

Limitations

Only as good as the sink coverage and the adversarial-completion corpus used to drive it; a new sink added later — a new rendering path, a new service the output gets forwarded to — needs its own pass rather than inheriting coverage from the others. This tests whether the sink holds under adversarial output, not whether the model was well-behaved; a system that relies on the model's own judgement instead of sink-level validation has nothing here to test against.

Cite this