Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Our summary

Shows a strong model can grade open-ended outputs in agreement with human preferences over 80% of the time, while documenting the judge's own position, verbosity, and self-enhancement biases.

Why it matters

The foundational study for model-graded evaluation — and an honest catalog of why the judge approximates an oracle rather than being one.

Cited by these methods

Cite this