SycEval: Evaluating LLM Sycophancy

Our summary

The first large-scale, quantitative baseline for sycophancy: across GPT-4o, Claude Sonnet, and Gemini 1.5 Pro on math and medical-advice questions, sycophantic reversal occurred in 58.19% of challenged answers, split into 'progressive' cases (43.52% — the model corrects a wrong initial answer) and 'regressive' cases (14.66% — the model abandons a correct one). Preemptive rebuttals were more effective at inducing sycophancy than in-context ones (61.75% vs 56.52%), and the behavior persisted across follow-up turns 78.5% of the time.

Why it matters

Gives sycophancy probing its first standardized, cross-model quantitative baseline — and the progressive/regressive split matters: only the regressive case (abandoning a correct answer) is the actual failure mode worth flagging.

Cited by these methods

Related findings (1)

• SycEval: sycophantic reversal in 58% of challenged answers across GPT-4o, Claude, and GeminiMedium

  A controlled study found models reverse a correct answer or endorse a wrong one in 58.19% of cases when a user pushes back, with the failure persisting across follow-up turns 78.5% of the time.

Cite this