Counting Ability of Large Language Models and Impact of Tokenization

Our summary

Studies how LLM counting degrades as the quantity grows and how tokenization shapes the error, arguing that counting needs reasoning depth scaling with the count — which transformers don't natively provide.

Why it matters

Explains why 'how many items are in this list' becomes unreliable as lists get longer, independent of the items themselves.

Related findings (1)

• Miscounting items in long listsLow

  Counts of items, occurrences, or matches in long inputs drift as list length grows.

Cite this