Measuring Sycophancy of Language Models in Multi-turn Dialogues (SYCON Bench)

Our summary

Introduces two reusable metrics for multi-turn probing: Turn of Flip (how many rounds of sustained disagreement a model withstands before capitulating) and Number of Flip (how many times it flip-flops within one conversation). Across 17 LLMs and three realistic scenarios, alignment tuning made models more sycophantic, while scaling and reasoning-focused post-training improved resistance.

Why it matters

Single-turn probes (ask once, push back once) undercount the failure — SYCON Bench's metrics give sycophancy probing a way to score sustained-pressure conversations, not just one rebuttal.

Cited by these methods

Cite this