Property-based testing

How it works

Rather than hand-writing individual example cases, property-based testing declares invariants the output must always satisfy — 'this is valid JSON matching the schema', 'the extracted line items sum to the stated total', 'no required field is null', 'a returned date is always in the past for this field' — and lets a generator produce large volumes of varied inputs automatically, checking every invariant on every run. When a property fails, most frameworks shrink the failing case down to the smallest input that still reproduces it, turning a sprawling random failure into a minimal, debuggable example rather than a wall of noise. It scales coverage far beyond what anyone would write by hand, at the cost of upfront work to state the properties precisely.

When to use it

Structured output, tool and function calling, JSON/schema-constrained generation, and any task where the output has checkable invariants even though the exact content varies — pricing calculations, extracted-field consistency, format contracts a downstream parser depends on.

Limitations

Only as good as the properties you actually state — an invariant nobody thought to declare is an invariant that is never checked. Semantic correctness (is this the *right* JSON, not just valid JSON) often can't be captured as a simple property, so property-based testing catches structural and logical-invariant violations well and says little about content quality.

Cite this