How LLM Tokens Work — And Why They Explain Your AI Bill

June 23, 2026 · updated June 25, 2026 · How Claude Actually Works (part 3)

▶ Watch on YouTube & subscribe to The Stack Underflow

Here is the thing almost nobody internalizes when they first use an LLM: Claude never reads your words. It reads tokens — integers. Your prompt is chopped into pieces by a program called the tokenizer, each piece is mapped to an integer ID, and the model only ever sees that sequence of numbers. Every cost surprise you have had, every context limit you have hit, and the famous “it can’t count the r’s in strawberry” bug — they all trace back to this one fact.

If you are building anything on the Claude API, this is the most load-bearing piece of intuition you can have. The dollar signs on your invoice are literally a function of token count, and so are every hard limit and “weird” edge case in the system.

The one-sentence version: Text is split into tokens (chunks roughly three-quarters of a word on average), each token maps to a number in a fixed vocabulary, and you pay per token — in and out — so understanding tokens is understanding cost, limits, and a class of model bugs in one shot.

What a token actually is

A token is a chunk of text — often a word, but frequently a piece of a word, a space, a punctuation mark, or even a single character. The tokenizer is a fixed lookup table (the vocabulary) that maps those chunks to integer IDs. The model does all its math on the IDs. It never touches the original characters.

Your prompt:  "Claude reads tokenization differently."
                        ↓  tokenizer
Token tiles:  ["Claude"] ["reads"] ["token"] ["ization"] [" differently"] ["."]
                        ↓  vocabulary lookup
Integer IDs:  [ 15040 ]  [ 16303 ] [ 3642 ]  [ 2065 ]   [  35001  ]     [13]

Notice that “tokenization” became two tiles — the common prefix “token” and the suffix “ization”. That is sub-word tokenization working as designed.

Rough rules of thumb for English text:

These are approximations. Code, JSON, and non-Latin scripts tokenize differently — more on that below.

Why models use tokens instead of words or characters

Two extremes, both broken:

• Whole-word vocabulary: The vocabulary would need millions of entries to cover all words in all languages, and it would still fail on any word it had never seen during training.
• Single-character vocabulary: Sequences would be thousands of characters long for a short paragraph, burning model capacity on reconstructing how letters form meaning.

Sub-word tokenization is the engineered middle ground. The vocabulary has roughly tens of thousands of entries — enough to cover common words as single tokens and rare or invented words as composites of known pieces. Any text can be encoded, including words the model was never trained on, by gluing sub-word pieces together.

Vocabulary size comparison (illustrative, not exact figures):
  Characters only:  ~256 entries → sequences too long
  Full words only:  ~1,000,000+ entries → too sparse, breaks on new words
  Sub-word tokens:  ~32,000–100,000 entries → tractable + general

The tokenizer is trained separately from the model itself — typically using an algorithm called Byte Pair Encoding (BPE), which iteratively merges the most frequent character pairs in a training corpus until a target vocabulary size is reached.

One important caveat flagged in the official docs (docs.anthropic.com, 2026): tokenizers change between model generations. Claude Opus 4.7 and all subsequent models (Opus 4.8, Sonnet 4.6, Fable 5, Mythos 5) use a new tokenizer. For a fixed piece of text, this newer tokenizer produces up to 35% more tokens than the tokenizer used by earlier Claude models. If you are migrating from a pre-4.7 model and relying on token-count estimates, re-measure after migration.

The “strawberry” bug — and what it reveals

The prompt-engineering community discovered that LLMs struggle to count individual letters in a word. Ask Claude “how many r’s are in strawberry?” and older or smaller models frequently get it wrong. This is not a reasoning failure — it is a tokenizer artifact.

"strawberry"  →  tokenizer  →  ["str"] ["aw"] ["berry"]

The letters r, a, w, b, e, r, r, y are sealed inside those three tiles. The model’s attention mechanism operates on the tiles as atomic units. To count individual letters the model would have to reason about the internal character structure of each tile — an operation it was not explicitly trained to do, because training data is in tokens, not characters.

This is why character-level operations (counting letters, reversing a word, detecting palindromes) are harder for LLMs than you would expect for a “language model.” The model does not primarily work at the character level. It works at the token level.

Why this explains your bill

Every API provider, including Anthropic, prices per token — and counts both directions:

• Input tokens: everything you send — your system prompt, user messages, conversation history, retrieved documents, tool definitions. All of it.
• Output tokens: everything the model generates back. Output is priced several times higher per token than input.

Current pricing on the Anthropic API (docs.anthropic.com, June 2026):

MTok = million tokens. These are standard synchronous API rates; Batch API cuts both by 50%, and prompt caching can cut cached input reads to 10% of the base rate.

Your bill ≈ (input_tokens × input_price) + (output_tokens × output_price)
              └─ prompt + history + docs + tools    └─ the model's reply

This is why costs surprise developers:

• Long context is not free. Claude Opus 4.8 and Sonnet 4.6 have 1M-token context windows. Filling one costs money on every single call.
• Conversation history compounds. In a chat loop, each new turn resends the entire prior conversation as input. Turn 20 is paying for turns 1 through 19 again.
• Verbose output costs more than verbose input. Asking for a 2,000-word answer at $25/MTok output is more expensive than sending a 2,000-word prompt at $5/MTok input — by a factor of 5.

A worked cost example

Say you are using Claude Sonnet 4.6 (input $3/MTok, output $15/MTok). You send a 1,000-token prompt and receive a 500-token answer:

Input:   1,000 tokens × ($3 / 1,000,000)  = $0.003
Output:    500 tokens × ($15 / 1,000,000) = $0.0075
One call ≈ $0.01

Tiny for one call. Now multiply:

The pattern is always the same: token count × call count. One expensive call is rarely the problem. It is the per-call token count quietly multiplied by volume.

Languages tokenize differently

This is the “language tax” developers often miss. Most tokenizer vocabularies were built primarily on English text. Non-Latin scripts and lower-resource languages tokenize into more tokens for the same semantic content — because fewer whole-word entries were in the training corpus, so more characters fall back to byte-level or character-level tiles.

Same meaning, different token costs:
  "Hello, how are you?"      (English)     →  ~6 tokens
  "Bonjour, comment allez-vous?" (French)  →  ~8 tokens
  "مرحبا، كيف حالك؟"        (Arabic)      →  ~12 tokens
  "안녕하세요, 잘 지내세요?"  (Korean)      →  ~16 tokens

If your application serves multiple languages, factor this into your cost model. Users in markets where the dominant language tokenizes densely will generate larger bills per semantic unit.

How to reason about token cost in practice

• Count tokens, not words. Use Anthropic’s token counting endpoint before finalizing a prompt design. The Python SDK has client.messages.count_tokens().
• Trim the context to what is needed. Every “just in case” paragraph is paid for on every call — even if the model never attends to it.
• Watch history growth in multi-turn applications. Prune, summarize, or drop old turns before they compound.
• Constrain output length when you do not need an essay. The max_tokens parameter is your budget cap per call.
• Cache the stable prefix. If you reuse the same system prompt or document across many calls, prompt caching drops the cost of cached reads to 10% of the base input rate. A large stable system prompt cached at $0.50/MTok (Opus 4.8 cache-read rate) instead of $5/MTok pays back the 1.25x cache-write cost after a single cache hit. See Prompt Caching: Cut Your AI Bill for the full mechanics.
• Prefer Haiku for high-volume, simple tasks. At $1/$5 MTok, Haiku 4.5 is five times cheaper than Opus 4.8. Match model capability to task complexity.

Common misconceptions

“The model reads my text.” No. The model reads token IDs — integers. Your text is transformed by the tokenizer before the model ever sees it. The characters you typed are not directly accessible to the model’s attention mechanism.

“One word equals one token.” Common words are often one token, but long or rare words split into multiple tokens. Punctuation, spaces, and line breaks are tokens too. Code and structured data (JSON, XML) are often more token-dense than equivalent prose.

“Only output costs money.” Both input and output are billed. Input is cheaper per token but there is typically much more of it, especially once you account for system prompts, conversation history, and retrieved context.

“The context window is free until you hit the limit.” The capacity exists regardless of use. But using it costs tokens on every call at the full per-token rate. A 1M-token context window is not a free buffer — it is a very large, very expensive bucket.

“The tokenizer is the same across all Claude models.” It is not. Claude Opus 4.7 introduced a new tokenizer, and all subsequent models (Opus 4.8, Sonnet 4.6, Fable 5, and beyond) use it. The same text can produce up to 35% more tokens on these models than on pre-4.7 Claude models. Always re-benchmark token counts after a model migration (docs.anthropic.com, 2026).

Frequently asked questions

How many tokens is a typical page of text? Roughly 500–800 tokens per page of prose in English, depending on vocabulary and formatting. Dense technical writing with many rare terms will tokenize toward the high end; conversational text toward the low end.

Why do code and JSON sometimes cost more tokens than they look? Symbols, indentation characters, brackets, and braces each tokenize separately. A deeply nested JSON object can be significantly more token-dense than its character count suggests. Minimising whitespace in prompts that pass structured data can reduce token count.

Does the system prompt count against my bill? Yes. The system prompt, tool definitions, and any retrieved context are all input tokens billed on every call. A 2,000-token system prompt sent 100,000 times a month is 200M input tokens — worth optimizing.

Is there a way to avoid resending the same big context every call? Yes — prompt caching lets you write a stable prefix to cache once and read it back at 10% of the base input rate. A cache write costs 1.25x the base input rate (for the 5-minute TTL) or 2x (for the 1-hour TTL). The break-even is one cache hit for the short TTL, two hits for the long TTL. Details are in Prompt Caching: Cut Your AI Bill.

Does the tokenizer affect model reasoning quality, not just cost? Yes. The new tokenizer in Claude Opus 4.7+ was designed alongside the model and contributes to improved performance. The tradeoff is that the same text costs more tokens on the new tokenizer. Anthropic’s docs describe this change explicitly.

What is the best way to measure token count before sending a request? Use the Messages API’s token counting endpoint: POST /v1/messages/count_tokens. Pass the same model, system, messages, and tools you intend to send. The response gives you the exact input_tokens count before you commit to the full request.

Where this fits in the series

Tokens are the atom of everything in this series. Once you see that every limit, every cost, and every edge case bottoms out at token count, the rest of the course clicks into place.

The immediately relevant next steps:

• Context window: Tokens fill a fixed bucket. How the Context Window Works explains what happens when it fills, how the model attends to a million tokens, and why position in the window matters.
• Cost reduction: Prompt Caching: Cut Your AI Bill shows how to drop the cost of repeated context to 10 cents on the dollar.
• The mental model that ties it together: The Claude Stack Mental Model shows where tokenization sits in the full stack — from raw text all the way to an agent calling tools.
• Where your tokens and dollars actually go: Where Your AI Tokens and Dollars Go breaks down a production request end-to-end.

Browse all tutorials to follow the full series.