GenAI and LLM interview question · Tokenization · junior level
Why are LLMs bad at counting letters or reversing a word?
Free to read in full. This is 1 of the 150 questions in the GenAI and LLM Engineering interview pack, with 173 diagrams. Get the pack, ₹999 · $20
Why it is asked
A classic check that you understand the model sees token ids, not characters, and that you know practical workarounds.
The answer
LLMs are bad at letter tasks because they never see letters: the tokenizer turns a word into a few token ids, and the model must remember from training which letters are inside each token.
Take 'strawberry'. With OpenAI's o200k_base tokenizer (used by GPT-4o) it becomes three tokens, 'st', 'raw' and 'berry', with ids 302, 1618 and 19772. With cl100k_base it is 'str', 'aw', 'berry'. To the model these are three arbitrary numbers. Nothing in id 19772 says 'two r letters'. The model can only answer 'how many r letters?' if it learned the spelling of each token from text where words were spelled out, and then added the counts correctly. That is indirect, so it fails often, especially for rare words.
Reversing a word is harder still: the model must spell each token, reverse the letters, and then write output whose tokens cut the word at completely different places.
The same cause explains other weaknesses: arithmetic on long numbers (digits are grouped into tokens in uneven ways), rhymes, anagrams, and character-level edits.
Workarounds:
- Ask the model to spell the word with separators first ('s-t-r-a-w...'), so each letter becomes its own token, then count.
- Give it a tool: a code interpreter does 'strawberry'.count('r') = 3 exactly.
- For products, never rely on the model for exact character work such as validating IDs or lengths; do it in code.
Newer reasoning models do better because they spell words out step by step, but it is still the tokenizer that makes the task hard.
What the interviewer listens for
- The model sees token ids, not letters
- 'strawberry' is 3 tokens in o200k_base: st, raw, berry
- Letter knowledge per token is learned indirectly
- Fix: spell out with separators, or use code tools
Follow-ups the interviewer may ask
- Why are LLMs weak at multiplying large numbers? Hint: digits are chunked into tokens unevenly, and carrying needs exact step-by-step work.
- How do you make letter counting reliable? Hint: call a code tool or force one token per character.
- Would a character-level model fix this? Hint: yes for spelling, at the cost of much longer sequences.
Common mistakes
- Saying the model is just 'bad at math'
- Assuming the word is one token or one token per letter
- Trusting the model for exact string validation in production
Written by Roni Das, who builds distributed systems and AI systems for a living.
Learn it properly
This is taught step by step in the course lesson Splitting Text Into Pieces: Why Search Works Better on Small Chunks.
One question is not enough for the interview.
This is 1 of the 150 questions in the GenAI and LLM Engineering pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (173 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- Write out scaled dot-product attention with the shape of every tensor. (mid)
- How does speculative sampling keep exactly the target model's distribution? (senior)
- What is GRPO, and how do verifiable rewards train reasoning models? (senior)
- Design a system that answers questions from 300-page contracts accurately. (staff)
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)