In plain words
AI language models don't read letters or whole words. They read tokens: chunks of text that might be a full word ("chai"), part of a word ("believ"), a space plus a word, or a single symbol ("!").
Each token is turned into a number, and the model only ever works with those numbers. When it replies, it writes one token at a time, then turns the numbers back into text for you.
Tokens matter to you in three ways. Models have a limit on how many they can handle at once (the context window). Paid AI services charge per token. And speed is measured in tokens per second. In English, one token is roughly three quarters of a word, so 100 tokens is about 75 words.
Everyday example
Coins at the toll booth
Think of a highway where you can only pay in coins of fixed sizes. A ₹47 toll gets paid as ₹20 + ₹20 + ₹5 + ₹2. Text works the same way: a long word gets paid in a few common chunks the model already has, like "Un" + "believ" + "ably". Common words fit in one coin.
Where the example breaks: Coins have fixed values everyone agrees on. Tokens don't: each model has its own set of chunks, so the same sentence splits differently in different models.
How it actually works
Turning text into tokens is called tokenization. It happens before the model sees anything, and the model's vocabulary of tokens is fixed when it is trained.
Build a vocabulary from text
Before training, a tokenizer studies a large sample of text and keeps the chunks that appear most often. A popular method, byte pair encoding (BPE), starts from single characters or bytes and keeps merging the most frequent neighbouring pairs until it has a set number of tokens.
Split new text into known chunks
When you type a message, the tokenizer replays the merges it learned, so the text ends up as chunks from its vocabulary. Common words become one token. Rare words, names, typos and numbers break into several.
Map each token to an ID
Every token in the vocabulary has a number, its ID. The model receives a list of IDs and turns each one into an embedding, a vector it can calculate with.
Generate token by token
The model predicts the next token ID, adds it, and repeats. The IDs are decoded back into text as they stream onto your screen, which is why replies appear in small bursts.
- Rule of thumb (English)
- 1 token is about 4 characters, or about 3/4 of a word
- Vocabulary sizes
- GPT-2 used 50,257 tokens. Meta's Llama 3 uses about 128,000
- Common method
- Byte pair encoding, adapted for language models in 2015
- Why it costs
- AI services usually price per million tokens, counting your input and the model's output separately
Many languages need more tokens than English for the same meaning, because tokenizers were built mostly from English text. Research has found some languages need several times as many, including some Indian languages. That makes them slower and costlier to use, though newer tokenizers with bigger vocabularies narrow the gap.
Questions people ask
How many words is 1,000 tokens?
About 750 words of ordinary English. Code, numbers, names and many non-English languages use more tokens per word, so the count varies.
Why do AI models struggle to count letters in a word?
Because they never see letters, only tokens. A word like "strawberry" may arrive as two or three chunks, so the model has to infer the spelling rather than read it.
What is a token limit?
The most tokens a model can handle in one go, counting your message, any files, the conversation so far and its reply. It is the model's context window.
Are tokens the same in every AI model?
Is a token the same as a crypto token?
No. In AI, a token is a piece of text. Crypto tokens are digital assets on a blockchain. The words just happen to match.