Output-handling / downstream-injection testing

How it works

When model output is passed unsanitized into a shell, a SQL query, a browser, a rendering pipeline, or another LLM, the vulnerability class is a familiar one — injection, SSRF, XSS, remote code execution — just arrived at by a new route. Testing this means exercising the actual downstream sink with adversarial completions, whether elicited through prompt injection or written directly as red-team input, and asserting invariants at the boundary: every output escaped, validated against a schema, or checked against an allowlist before it's used, exactly as if it came from an anonymous user. This applies equally to natural-language output and to code the model writes and something else executes — the model's fluency is not a safety property of the sink.

When to use it

Any system that renders, executes, queries, or otherwise acts on model output without a human reviewing it first — code execution, database queries, markdown/HTML rendering, shell commands, calls to other services.

Limitations

Only as good as the sink coverage and the adversarial-completion corpus used to drive it; a new sink added later needs its own pass. Fixing this at the model layer (asking it to 'be careful') is not a substitute for validating at the sink.

Cite this