Asking for oracles

How it works

AI test generation typically produces inputs and steps. In practice, the hard part of testing is usually recognising whether what happened was a problem. Give the model the feature description, a few real outputs and the test you plan to run, then ask specifically for oracles: things that must always hold, such as totals that reconcile or a created record that can be retrieved; previous versions or comparable products to diff against; claims in documentation the behaviour should match; and relations under transformation, such as shuffling the input and getting the same sorted result, or filtering and then unfiltering. For each oracle, ask what failure it would catch and what it would miss. The output is a list to choose from, not a verdict.

When to use it

Features with no written expected results; outputs too many or too complex to check by eye, such as calculations, reports and generated documents; and before automating, to decide what a check should actually assert.

Limitations

The suggestions sound plausible but aren't verified, and an invented invariant can be false for your product and send you chasing non-bugs. Domain rules — regulation, contracts, local conventions — are where the model's oracles are weakest and where they matter most. It also leans towards generic consistency checks and under-proposes oracles that depend on product history or knowledge of users.

What you still own

Deciding which oracles are actually true for this product, and confirming each one against a source or an expert before relying on it.

Related evals

Cite this