TruthfulQA: Measuring How Models Mimic Human Falsehoods

Our summary

A benchmark of questions where humans commonly hold misconceptions; models often answer with the same imitative falsehoods learned from training text, and — strikingly — larger models can be less truthful. Separates being informative from being truthful.

Why it matters

Shows fabrication isn't random noise: models reproduce popular human falsehoods, which is exactly when a confident wrong answer is most persuasive.

Cited by these methods

Related findings (2)

• Fabricated citations and referencesHigh

  Models invent plausible-looking but non-existent papers, authors, DOIs and URLs.

• Fabrication instead of admitting uncertaintyHigh

  Asked about something unknown or non-existent, models invent an answer rather than saying 'I don't know'.

Cite this