BBQ: A Hand-Built Bias Benchmark for Question Answering

Our summary

A hand-built benchmark probing social bias in QA across nine dimensions: models fall back on stereotypes when context is under-specified, and are more accurate when the correct answer happens to match a stereotype.

Why it matters

A rigorous, counterfactual approach to bias probing — vary only the protected attribute and watch the answer change.

Cited by these methods

Related findings (1)

• Name-based demographic bias in outputsHigh

  Swapping only a name (signalling gender or ethnicity) changes evaluative outputs like screening or sentiment.

Cite this