In plain words
A language model writes its answer in tokens, small pieces of text about three quarters of a word each. Tokens per second measures how quickly those pieces come out. It is the speedometer of AI.
There are two speeds to know. One is your speed: how fast the words appear on your screen. The other is the speed of the whole server: how many tokens it writes per second for everyone it is serving at once. A system can be fast overall and still feel slow to each person.
For scale, people read about 238 words a minute on average, which works out to roughly 5 tokens a second. A model writing faster than that stays ahead of your eyes. Speed matters most when AI writes long code or documents, or works through many steps on its own as an agent.
Everyday example
The photocopy shop near college
A photocopy shop near a college has one big machine. For you, what matters is how fast your own notes come out. For the owner, what matters is pages per minute across all customers, so he runs several orders through together. Each order comes out a little slower, but the shop prints far more in a day. Tokens per second has the same two sides.
Where the example breaks: A photocopier's speed is fixed by the machine. A model's speed also depends on its size, the chip, how long the conversation already is and how many others share the chip at that moment.
How it actually works
Writing speed is set mostly by memory, not maths: to produce each token, the chip must read through the model's weights again.
One token at a time
A model cannot write token ten before token nine, because each one depends on all the ones before. So speed for one person is roughly one divided by the gap between tokens.
Memory is the speed limit
For each new token the chip reads the model's billions of weights from memory, while doing comparatively little maths. Faster memory and smaller models write faster, which is one reason quantization speeds things up.
Many users, one read
Servers read the weights once and use them for many users' next tokens at the same time, called batching. Total tokens per second climbs as more requests share the chip, until the chip is full; after that, each user just waits longer.
Ways to go faster
Speculative decoding lets a small, fast model guess several tokens that the big model then checks in one go; a 2022 paper reported 2 to 3 times faster output with identical results. Chips built for inference are another route.
- Rough size
- 100 tokens is about 75 English words (OpenAI's rule of thumb)
- Human reading speed
- About 238 words a minute, roughly 5 tokens a second (a 2019 review of 190 studies)
- Two measures
- Tokens per second for one user, and total throughput across all users
- Depends on
- Model size, chip and memory speed, prompt length, and how many requests share the chip
Speed per user and total throughput pull against each other, so providers choose a balance. That choice also shapes inference cost.
Questions people ask
What is a good tokens per second speed?
It depends on the job. For reading along in a chat, anything comfortably above reading speed, about 5 tokens a second, feels fluent. For long code or for AI agents that chain many steps, much higher speeds save real time. There is no fixed standard.
What is the difference between tokens per second and latency?
Latency is how long you wait, especially for the first word. Tokens per second is how fast the rest arrives once it starts. A model can be slow to start and then write quickly, or start instantly and then crawl.
Why do bigger models write slower?
Every new token means reading all of the model's weights from memory. A model with ten times more parameters has roughly ten times more to read per token, so on the same chip it writes more slowly. Mixture-of-experts models get around this by using only part of their weights for each token.
Does a faster GPU always mean more tokens per second?
Not always. Writing is usually limited by memory speed, not by how many sums the chip can do. A GPU with faster HBM memory often gains more for chatbots than one with extra raw maths power. Software and batching matter a lot too.
What does throughput mean in AI?
Throughput is the total amount of work a system gets through, usually counted as tokens per second across all users together. It tells a provider how many people one set of chips can serve, which drives cost. Your own speed is a separate number.
Further reading
- LLM Inference Benchmarking: Fundamental ConceptsNvidia Technical Blog, 2025
- LLM inference: prefill, decode and batchingTogether AI, 2026
- What are tokens and how to count them?OpenAI, 2026
- Most comprehensive review to date finds the average person's reading speed is slower than commonly thoughtBritish Psychological Society Research Digest, 2019
- Fast Inference from Transformers via Speculative Decoding (Leviathan, Kalman, Matias)arXiv, 2022