Why AI models miscount letters in words like 'strawberry'

Ask several AI language models how many times the letter R appears in the word “strawberry” and you will sometimes get an answer that is wrong — often two, when the true count is three. This is not a sign that the model is careless or unintelligent. It happens because most language models never see the word as a sequence of individual letters at all. They see it as a small number of chunks called tokens, and counting a letter means answering a question about a level of detail the model was never shown directly.

The demonstration

The “strawberry problem” became a well-known way to probe this weakness because the word is common and the correct answer is trivial for a person to check by reading letter by letter. Put the question to a range of models and the failure shows up unevenly: some answer correctly, some do not, and the same model can sometimes get it right on one phrasing and wrong on another. Model builders have worked on this class of problem since it drew attention, and newer models tend to do better on it than older ones. But treat any claim that a specific current model “always” gets it right, or “always” fails, with caution — the reliable pattern is the class of error, not a fixed scoreboard of which model passes today.

Why it happens

Language models do not process text one letter at a time. Before a model sees any input, a step called tokenisation breaks the text into tokens — chunks that are often whole common words, word fragments, or a few characters, chosen because they let the model handle a huge vocabulary efficiently. The word “strawberry” is unusual enough that it typically becomes more than one token, but crucially still far fewer tokens than it has letters. Once tokenised, the model reasons over those chunks and the patterns of meaning associated with them, not over the individual characters inside each chunk.

Asking “how many R’s are in strawberry” is therefore asking the model to report on something it has to reconstruct rather than something it can simply read off. The model can often get close by drawing on patterns it absorbed during training, or by reasoning step by step if prompted, but the direct path — looking at each letter in turn — is not how the underlying architecture represents the word. NIST’s work on AI measurement and Stanford HAI’s research on evaluating model capabilities point to the same idea: a model’s errors often trace back to a mismatch between the task being asked and the representation the model actually operates on, not to a general lack of intelligence.

What it reveals

The letter-counting stumble is a specific case of a wider pattern researchers describe: these models are strong at tasks involving meaning, association and fluent language, and comparatively weak at tasks requiring exact, character-by-character precision. The same root cause — reasoning over tokens rather than raw symbols — helps explain why models can also fumble precise arithmetic on longer numbers, misspell an unusual word while otherwise writing fluently, or miscount items in a list. In each case, the task demands exact symbol-level tracking, which sits awkwardly against how these systems represent text internally.

This does not mean the models “don’t understand” language in any simple sense — researchers are careful to say that what a model understands, and whether that word even applies, is a genuinely contested question rather than a settled one. What is better established is narrower and more useful: certain task shapes reliably strain these systems, regardless of how fluent their output looks elsewhere.

The takeaway

The practical lesson is not to distrust AI models generally, but to learn the shape of tasks where they are known to be weak: exact character counts, precise multi-digit arithmetic, strict spelling of unusual words, and similar symbol-level precision tasks. For anything in that shape, check the output — with a calculator, a spellchecker, or by counting yourself — rather than taking a fluent-sounding answer as correct. Fluency and accuracy are not the same thing, and knowing where they come apart is more useful than memorising which model currently passes the strawberry test.