The Curious Case of Neural Text Degeneration

Our summary

The paper that diagnosed degenerate repetition in neural text generation and introduced nucleus (top-p) sampling, showing that maximum-likelihood decoding produces bland, looping text.

Why it matters

The foundational account of repetition and looping as a decoding-level failure mode, and the origin of the standard mitigation.

Related findings (1)

• Repetition and degeneration loopsLow

  Under certain prompts or long generations, models fall into repeating phrases or degenerate text.

Cite this