MediumTool useReviewer-confirmedPublished

Format-constraint violations under strict schemas

Asked for strictly-formatted output (e.g. JSON to a schema), models emit invalid or extra content.

Published June 26, 2026

Reproducibility: Sometimes
Severity: Medium
Confidence: Reviewer-confirmed

Details

Under strict structured-output requirements, models add prose, omit required fields, or produce malformed JSON, especially as schema complexity grows. A core reliability issue for tool/function calling and pipelines.

Found with

🔬 Property-based testing

Validate every output against the JSON schema.

🔬 Boundary & edge-case testing

Failure rate rises with schema depth.

🔬 Unit testing the deterministic scaffold

A schema-validation unit test rejects the malformed output deterministically, before it ever reaches a parser.

🔬 Smoke testing in CI/CD

A daily CI smoke run on a canonical structured-output prompt catches the deploy where a model update starts breaking the schema.

Evidence

Requested pure JSON; output wrapped in ```json fences with a trailing comment, breaking parsers.

Illustrative example — see the linked reference for the documented evidence.

Affected versions

OpenAI · gpt-4oGoogle · gemini-2.0-flashMeta · llama-3.3-70bMistral · mistral-large-2

References

Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models

Tool use Agents

Source: https://arxiv.org/abs/2408.02442

Cite this

Qlarify Labs. (2026). Format-constraint violations under strict schemas. Retrieved from https://labs.qlarify.fi/findings/strict-json-violations