Faith and Fate: Limits of Transformers on Compositionality

Our summary

Probes the limits of transformers on compositional tasks including multi-digit multiplication, showing accuracy collapses as a problem requires more sequential sub-steps — the model pattern-matches rather than executing a reliable algorithm.

Why it matters

Grounds multi-digit arithmetic errors in a structural limit on compositional computation, not a shortage of training examples.

Related findings (1)

• Errors in multi-digit arithmeticMedium

  Without tool use, models make systematic errors on multi-digit multiplication and long arithmetic.

Cite this