In plain words
When you ask a chatbot a question, there is a short pause before the first word appears. That pause is latency. It is made of several waits added up: your message travelling to a data center, waiting in a queue behind other users, and the model reading your whole prompt.
Engineers watch two numbers most. Time to first token is how long until the first word shows up. After that, the rest of the answer streams in piece by piece, and the gap between pieces decides how long the whole reply takes.
Latency matters because people notice delay quickly. Under about a tenth of a second feels instant; past ten seconds, attention drifts. Voice assistants need very low latency to feel natural, while a long research report can take minutes and still be worth the wait.
Everyday example
Chai for the whole group
You order chai for six friends. First the stall owner has to hear you over the crowd, then finish the two orders before yours, then boil the milk. That wait until the first cup is poured is time to first token. Then he pours cup after cup, one at a time. How fast he pours decides when the last friend gets theirs. Latency is both waits added together.
Where the example breaks: You cannot drink chai that is still being poured, but you can read an answer that is still being written. AI apps hide much of the wait by streaming each word as soon as it exists.
How it actually works
For a language model, end-to-end latency is the time to first token plus the time to generate every token after it.
Network and queue
Your request crosses the internet, sometimes thousands of kilometres, then may wait for a free chip. At busy times, waiting in the queue can be the biggest part of the delay.
Reading the prompt
The model first processes your whole prompt in one parallel pass, called prefill. Long prompts, such as a pasted document, take longer here, which pushes back the first token.
Writing the answer
Then the model writes one token at a time, each step reading through the model's weights again. The average gap between tokens, called inter-token latency, sets how fast the text streams. See tokens per second.
Thinking first
Reasoning models produce hidden reasoning before the visible reply, so you can wait many seconds, or minutes, for the first word even on fast chips.
A balancing act
Serving more users on the same chips raises total output, but each person waits a little longer. Providers tune that balance. Tricks like speculative decoding, where a small model drafts tokens and the big model checks them, cut the wait without changing the answer.
- Feels instant
- About 0.1 seconds (Nielsen Norman Group, drawing on research from 1968 and 1991)
- Attention limit
- About 10 seconds, after which people tend to drift to other tasks
- Main AI measures
- Time to first token (TTFT), inter-token latency, and end-to-end latency
- Speculative decoding
- 2 to 3 times faster in a 2022 paper that described it, with identical output
Latency is about how long one person waits; throughput is about how much work the whole system gets through. Both matter, and they pull against each other.
Questions people ask
What is time to first token?
Time to first token, or TTFT, is how long you wait from sending a message until the first piece of the answer appears. It includes the network trip, any queue, and the model reading your prompt. Long prompts and reasoning models usually make it longer.
Why is ChatGPT slow sometimes?
Usually one of four things: many people using it at once, so requests queue; a long prompt or attached file that takes longer to read; a reasoning model thinking before it answers; or a slow internet connection. Shorter prompts and faster model options often reply sooner.
Why does AI type out answers word by word?
Because that is how it writes them. A language model produces one token at a time, each based on everything before it. Apps show each token as soon as it is ready, called streaming, so you can start reading long before the full answer is done.
What is the difference between latency and throughput?
Latency is how long one request takes, from your point of view. Throughput is how many tokens or requests the whole system handles per second across all users. Packing more users onto the same chips raises throughput but usually adds a little latency for each of them.
Is on-device AI faster than cloud AI?
It can start faster, because there is no trip over the internet and no queue. But phone and laptop chips are far less powerful, so on-device AI uses small models. For short tasks it often feels quicker; for big ones, cloud AI usually wins.