In plain words
Older language AI read a sentence the way we read a ticker: one word at a time, left to right, trying to remember what came before. Long sentences made it forget the beginning.
A transformer reads every word at the same time. For each word, it asks one question: which other words here help me understand this one? That trick is called attention. It lets the model connect "it" to "cat" even when ten words sit between them.
Because every word is handled in parallel, transformers can be trained on huge amounts of text using thousands of GPUs at once. That is why almost every large language model today is built on one. The T in GPT stands for transformer.
Everyday example
A WhatsApp group reading one message
Someone posts "She said yes!" in the family group. Nobody reads it alone. Everyone's eyes jump to the earlier messages: who is "she", what was the question? Each person pulls in the bits that matter and ignores the rest. A transformer does this for every word in a passage, all at once.
Where the example breaks: People in a group chat understand what is going on. A transformer only learns which words tend to go together and in what way. It works with patterns in numbers, not meaning the way we feel it.
How it actually works
Under the hood, a transformer turns text into numbers and then refines those numbers through a stack of identical layers.
Attention mixes in context
In each layer, every token builds three vectors called query, key and value. A token's query is compared with every other token's key. The better the match, the more of that token's value gets blended in. This is self-attention, and it runs in several parallel heads that each learn a different kind of link. In chatbot models, each token can only look at the tokens before it.
A small network refines each token
After attention, each token passes through its own small neural network (a feed-forward layer). This is where much of the model's stored knowledge seems to live.
Repeat, layer after layer
Attention plus feed-forward makes one block. Large models stack dozens of these blocks. Early layers tend to capture grammar, later ones capture more abstract relationships.
Predict what comes next
In a chatbot, the final layer scores every possible next token. The model picks one, adds it to the text, and runs again. This loop is next-token prediction.
- First described
- June 2017, in the paper "Attention Is All You Need"
- Who
- Eight researchers working at Google. Ashish Vaswani is listed first, but the paper says all contributed equally
- Built first for
- Translating between languages (English to German and French)
Transformers are not only for text. The same design now powers image models (vision transformers), speech systems and parts of many diffusion models.
Questions people ask
What does the T in ChatGPT stand for?
Transformer. GPT means generative pre-trained transformer: a model that generates text, was pre-trained on a large body of text, and is built on the transformer design.
Is a transformer the same as an LLM?
No. A transformer is the design, like a type of engine. A large language model is a specific trained model built with that design, like a particular car. Most LLMs today are transformers.
Who invented the transformer?
A team of eight researchers working at Google described it in the 2017 paper "Attention Is All You Need". Ashish Vaswani is listed first, and the paper notes the author order is random and the work was shared. Attention itself came earlier, in 2014, in translation models.
Why did transformers replace older models?
Older recurrent networks read one word at a time, which was slow to train and forgot long-range context. Transformers look at all words at once, so they train in parallel on far more data and handle long passages better.
Do transformers have a limit on how much they can read?
Yes. Attention compares every token with every other token, so cost grows quickly with length. That limit is the context window. Early transformer models handled a few hundred tokens. Engineering tricks have pushed some of today's models past a million.