In plain words
Before a chatbot can answer anything, its model goes through pre-training. It is shown a huge pile of text: web pages, books, articles, code. Then it plays one game over and over: cover the next word, guess it, check.
Any sentence can become practice. "The monsoon reaches Kerala in June" gives several free questions: what comes after "The monsoon"? After "reaches"? Nobody has to write answers, because the real text already holds them.
Doing this trillions of times teaches the model grammar, facts, styles and a surprising amount of reasoning. It takes thousands of powerful chips running for weeks or months, which is why only a few organisations pre-train the largest models.
The result is a base model: knowledgeable, but it only continues text. Later, shorter stages of fine-tuning turn it into a helpful assistant.
Everyday example
Years of reading before medical college
A doctor does not start with surgery. First come years of school: reading, maths, biology, a broad base of knowledge picked up from many books. Only then come a few focused years of medical training. Pre-training is the school years for an AI: broad, long and expensive. Fine-tuning is the short specialisation that comes after.
Where the example breaks: A student learns from teachers who explain things. Pre-training has no teacher and no explanations, only the next word to guess, trillions of times. Whatever the model learns is whatever helps that guess.
How it actually works
Pre-training is where almost all of a model's knowledge and most of its training cost come from.
Gather and clean the data
Teams collect text from the public web, books, code and other sources, then filter out duplicates, spam and low-quality pages. GPT-3's 2020 mix was 60% filtered web crawl, plus books, Wikipedia and a curated web set. Newer models also mix in images, audio and video.
Predict, compare, adjust
The model (transformer) predicts every next token in a batch of text. The gap between its guesses and the real tokens is measured by a loss function, and gradient descent nudges billions of parameters to shrink it. This is self-supervised learning: the data labels itself.
Run it at enormous scale
Training runs across thousands of GPUs in parallel. Meta trained Llama 3 on two clusters of about 24,000 GPUs each. Bigger models and more data have reliably given better results, a pattern called scaling laws.
Hand over to post-training
The finished base model goes through instruction tuning, RLHF and other post-training steps. These use far less data but shape how the model behaves.
- Named
- GPT stands for generative pre-trained transformer, after OpenAI's June 2018 paper on generative pre-training
- GPT-1 data
- BooksCorpus: over 7,000 unpublished books
- Growth
- 300 billion tokens (GPT-3, 2020) to over 30 trillion (Llama 4, 2025): about 100 times in five years
- Openness
- As of October 2026, most leading labs no longer publish exact data sizes for their newest models; Meta's Llama figures are among the few public ones
Questions people ask
What is the difference between pre-training and fine-tuning?
Pre-training is broad and huge: the model learns general language and knowledge from trillions of tokens of raw text. Fine-tuning is narrow and small: it adjusts an already pre-trained model with a much smaller, carefully chosen set of examples for a task or behaviour.
What does "pre-trained" mean in GPT?
GPT means generative pre-trained transformer. "Pre-trained" says the model first learned from a large general body of text before being adapted for specific uses. OpenAI's 2018 paper combined this pre-training idea with the transformer design.
What data are AI models pre-trained on?
Mostly public web pages, plus books, encyclopaedias, code, academic papers and, increasingly, images, audio and video. Exact mixes are often secret. Whether using copyrighted material this way is allowed is being argued in courts (AI and copyright).
What is a base model?
A base model is what comes out of pre-training, before any instruction tuning. Ask it a question and it may just continue the text, for example with more questions. It knows a lot but has not yet learned to act like an assistant.
How long does pre-training take?
For the largest models, weeks to months on thousands of chips running non-stop. Exact times and costs are rarely published. Small models can be pre-trained in days on far less hardware.