Flaky test diagnosis

How it works

Finding flaky tests is a statistics problem: a test that passes and fails on the same commit, or flips more often than its neighbours, is flaky — though the cause may lie in the environment or the product rather than the test. That part is well established and works from CI history alone. Diagnosing why is harder, and is where a model helps: given the test code, the code under test and a sample of passing and failing logs, it looks for the common causes — fixed waits and timing assumptions, shared state between tests, test order dependence, unseeded randomness, time zone or date dependence, and external services. It suggests a specific fix for the most likely cause. Quarantining flaky tests while they're investigated keeps them from hiding real failures.

When to use it

Suites where people have started rerunning failures by habit; before trusting a suite to gate releases; and when one or two notorious tests consume a disproportionate share of investigation time.

Limitations

Some flakiness reflects a real race condition or resource leak in the product, and "fixing" the test by adding a retry or a longer wait hides a genuine bug. Diagnosis from code and logs misses causes that live in the infrastructure — noisy neighbours, slow disks, network jitter. Quarantine without follow-up becomes permanent, and the suite quietly loses coverage.

Before you reach for it

Wait for conditions rather than fixed times, and give each test its own data, which removes the commonest causes before any diagnosis is needed.

What you still own

Deciding whether the flakiness is in the test or the product, and making sure quarantined tests come back.

Cite this