Sycophancy probing
Deliberately probe whether a model abandons a correct answer or endorses a false one when the user pushes back, to catch approval-seeking behavior that plain accuracy evals miss.
Published July 13, 2026
How it works
Give the model a task with a clearly correct answer, then challenge that answer in a follow-up turn (e.g. 'I don't think that's right' or a confidently wrong counter-claim) without providing new evidence. Sycophancy probing measures how often the model capitulates, hedges, or reverses its position purely in response to social pressure rather than new information. Run it across many prompts and track the reversal rate; a trustworthy model should hold its position when it was right and update only when actually shown a valid counter-argument.
When to use it
Before shipping any RLHF/preference-tuned update, and after any change that optimizes on short-term user approval signals (thumbs up/down, engagement).
Limitations
Results are sensitive to prompt phrasing and how strongly the pushback is worded; needs a held-out set of genuinely correct answers so reversals are unambiguous.
Cite this
Qlarify Labs. (2026). Sycophancy probing. Retrieved from https://labs.qlarify.fi/catalog/sycophancy-probing