Black October, explainedAppsGuidesAI DictionarySmall FixesOn RecordAboutFollow the lab

Tokenization

Also called tokenizer, tokenisation, BPE

Tokenization is how an AI cuts text into small pieces called tokens. Common words become one piece, rare words get split into several.

Inside language modelsTechnical

In plain words

An AI model cannot read letters the way we do. Before it sees anything, a small program called a tokenizer cuts the text into pieces and swaps each piece for a number.

The tokenizer does not use a dictionary. It learns its pieces by reading lots of text and noticing which letters keep turning up together. "c" and "h" sit side by side so often that they become one piece, "ch". Then "ch" and "a" join, and so on, until common words like "chai" are a single piece.

Rare words never get merged that far, so they stay in bits. "Thiruvananthapuram" comes out as seven pieces. That is why the same sentence can cost more tokens in Hindi or Tamil than in English, and why a chatbot can stumble when asked to count the letters in a word.

Everyday example

Shorthand in a class notebook

A student taking notes writes "bcz" for because and "govt" for government, the words that come up every lecture. Rare words, like a professor's surname, get written out letter by letter. Over a term, the shorthand grows from what she hears most. A tokenizer builds its pieces the same way: the most frequent combinations get their own short symbol.

Where the example breaks: The student knows what her shorthand means. A tokenizer only counts which pieces appear together. It has no idea that "chai" is a drink; working that out is the model's job later.

Go deeper

How it actually works

Most modern language models use a method called byte pair encoding (BPE), or close relatives of it such as SentencePiece.

  1. Start from the smallest units

    Training begins with single characters, or with the 256 possible bytes so that any script or emoji can be written. Text is first roughly split at spaces and punctuation.

  2. Merge the most common pair, again and again

    The trainer counts every pair of neighbouring pieces across a huge sample of text, joins the most frequent pair into a new piece, and repeats. Each merge is saved as a rule. This stops when the vocabulary reaches a chosen size.

  3. Apply the rules to new text

    To tokenize a fresh sentence, the tokenizer splits it into bytes and replays the saved merges in order. Each final piece is looked up in the vocabulary and replaced by its ID number. The model then turns each ID into an embedding.

  4. Why the vocabulary size matters

    A bigger vocabulary means fewer tokens per sentence, so more fits in the context window. OpenAI's tokenizer for GPT-4 has about 100,000 pieces; the one for GPT-4o has 200,019. In our test, a Hindi greeting dropped from 20 tokens to 9 between the two, while its English version stayed at 6.

First used for language AI
2015, by Rico Sennrich, Barry Haddow and Alexandra Birch at the University of Edinburgh, for translation
Borrowed from
Byte pair encoding, a 1994 data compression method by Philip Gage
Rule of thumb
In English, one token is about four characters, or three quarters of a word (OpenAI)
Fairness gap
The same text can take up to 15 times more tokens in some languages than others (Petrov et al., 2023)

Each model family has its own tokenizer, so token counts for the same text differ between GPT, Claude and Gemini.

Questions people ask

What is the difference between a token and tokenization?

A token is one piece of text, like "chai" or "ization". Tokenization is the process of cutting text into those pieces and turning them into numbers. The tool that does it is called a tokenizer.

Why can't AI count the letters in strawberry?

The model never sees the letters. GPT-4o's tokenizer turns "strawberry" into three pieces: st, raw and berry. Counting the r's means recalling a spelling it was never shown directly. Models that reason step by step can do better by writing the word out letter by letter first.

Why does AI cost more in Hindi or other languages?

Tokenizers learn from training text that is mostly English, so English words become single pieces while words in other scripts are split into more of them. Since AI services charge and limit by tokens, the same message can cost more. Newer tokenizers with bigger vocabularies have narrowed the gap.

Is tokenization in AI the same as tokenization in payments?

No. In payments, tokenization swaps a card number for a stand-in code, so shops and apps never store the real number. In AI it means cutting text into pieces for a model. Same word, two unrelated ideas.

How can we see how text gets tokenized?

OpenAI publishes a free online tokenizer page and an open-source library called tiktoken. Paste any sentence and each token is shown in its own colour with its ID number. It is a quick way to see why some text uses more tokens than expected.

Further reading

Prepared by Mundhra Labs. Reviewed by Aditya Mundhra. Checked October 2026.