Self-consistency probing
Ask the same question multiple times (or multiple ways) and measure how often the answers agree.
Published June 26, 2026
How it works
A confident model that gives different answers to the same question is unreliable regardless of which answer is right. Sampling repeatedly and measuring agreement turns stochasticity into a quantitative reliability signal, and disagreement pinpoints questions the model doesn't actually 'know'.
When to use it
Reliability assessment; flagging low-confidence outputs; calibration studies.
Limitations
Consistent does not mean correct — a model can be reliably wrong.
Method yield
- Findings
- 3
- Versions spanned
- 5
- Yield score
- 8
Severity-weighted across the published findings below. Why we measure this →
Findings it surfaces (3)
Documented failures this method catches — the evidence it works.
- Self-contradiction within a single conversationLow
Models assert one fact and later assert its opposite within the same session.
How it found it: Re-ask within session; answers conflict.
Reasoning - Poor uncertainty calibration / overconfidenceMedium
Stated confidence does not track accuracy; models sound equally certain when right and wrong.
How it found it: High stated confidence on answers that vary across samples.
Hallucination - Inconsistent answers to semantically equivalent promptsMedium
Trivial rewordings of the same question yield materially different answers.
Reasoning
References & further reading
Cite this
Qlarify Labs. (2026). Self-consistency probing. Retrieved from https://labs.qlarify.fi/methods/self-consistency-probing