Distributional testing (KS test, Monte Carlo)

How it works

A non-deterministic system has to be judged statistically. By sampling repeatedly and treating the outputs as a distribution, you can ask sharper questions: has this version's output distribution shifted from the last one (a two-sample Kolmogorov–Smirnov test), does stated confidence track actual accuracy, how wide is the spread of a metric under resampling (Monte Carlo). It turns stochasticity from a nuisance into a measurable property — the natural lens for stability, longevity, and decay.

When to use it

Calibration studies; detecting distribution shift between versions; quantifying variance in a metric; any claim about stability rather than a single output — particularly after a silent provider-side model update, where behavior on any one prompt can look unchanged while the underlying output distribution has shifted.

Limitations

Needs enough samples to be statistically meaningful, which costs tokens proportional to sample size, and a shifted distribution tells you behaviour changed, not whether it changed for the better. Choosing the right statistic and threshold is itself a modelling decision, and an overly sensitive threshold turns routine variance into constant false alarms.

Cite this