Smoke testing in CI/CD

How it works

Before the expensive evals, a smoke suite answers one question on every deploy: is the system fundamentally alive? A few canonical prompts run end to end, a model-reachability check, a token-budget sanity check, one representative happy path per major feature — say, one summarization request and one tool-calling request for a system that does both. It is cheap, runs in CI on every commit, and exists to catch broken APIs, expired keys, and config regressions so they never reach a human reviewer — let alone production. Its value is being the first gate, not the deepest one: a smoke suite that starts asserting on answer quality has stopped being a smoke suite and started being a slow, under-powered eval.

When to use it

In CI/CD on every build, and as the first gate before slower probabilistic suites are worth spending tokens on; especially valuable right after a deploy, where it catches an expired key or a misconfigured environment variable before a single real user reaches the broken path.

Limitations

Shallow by design — a green smoke run proves the system is alive and roughly on-script, not that it reasons well; a smoke suite tuned too broad slows every build without adding real signal. Depth has to come from the evaluation and robustness methods downstream.

Cite this