In plain words
Labelling data by hand is slow and expensive. But the world is full of unlabelled data: books, websites, recordings, photos. Self-supervised learning finds a way to learn from all of it.
The trick is to turn the data into its own quiz. Take a sentence, cover one word, and ask the computer to guess it. The covered word is the right answer, so nobody had to write a label. Do this billions of times across billions of sentences and the computer picks up grammar, facts and how ideas connect.
This is how large language models like the ones behind ChatGPT start out: their first and biggest stage of training, called pre-training, is self-supervised. The same idea works for images and speech by hiding patches of a picture or slices of a sound.
Everyday example
Making your own fill-in-the-blanks
A student has a textbook but no workbook. So she copies paragraphs, covers a word in each with a sticky note, and tries to fill the gap. Lifting the note shows if she was right. The book is both the question paper and the answer key, and she can make endless practice from any page.
Where the example breaks: The student knows what the paragraph means. The model only gets better at predicting what fits the gap, which builds useful knowledge but also means it can produce text that fits well and is still false.
How it actually works
Self-supervised learning sits between supervised and unsupervised learning: the data has no human labels, but the method creates labels from the data itself.
Invent a pretext task
Designers pick a task whose answer is already in the data: guess a hidden word, the next word, a missing patch of an image, or the next slice of audio. Yann LeCun's description: predict any hidden part of the input from the parts that are visible.
Masked words
BERT (Google, 2018) hid 15% of the tokens in each passage at random and learned to fill them in, using text from books and English Wikipedia.
Next word
GPT-style models hide the future instead: every position is trained to predict the token that comes next. That is next-token prediction, and the real next token is the label.
Train like supervised learning
Once the labels exist, training is ordinary: a loss function scores each guess and gradient descent adjusts the model. The difference is that the labels cost nothing, so the data can be enormous.
Reuse what was learned
The task itself is rarely the goal. What matters is the knowledge the model builds while solving it, which is then adapted to real tasks through transfer learning or fine-tuning.
- Core idea
- Labels made from the data itself, not by people
- Landmark
- BERT, October 2018: hid 15% of tokens and predicted them
- Where it is used
- Pre-training of language models, speech recognition, image models
- Why it matters
- Removes the human labelling bottleneck, so models can learn from vast amounts of raw data
Questions people ask
What is the difference between self-supervised and unsupervised learning?
Both use data with no human labels. Unsupervised learning looks for structure such as groups. Self-supervised learning hides part of each example and trains the model to predict it, so it has a right answer to check against, just like supervised learning.
Is ChatGPT self-supervised?
Its base model is. The language model was first pre-trained by predicting the next word across huge amounts of text, which is self-supervised. It was then tuned with human-written examples and human feedback to become a helpful chatbot.
Why not just label all the data by hand?
Because the amounts are far too large. Modern models learn from more text than any team could ever label. Self-supervised learning gets a training signal from every sentence for free, which is what made today's large models possible.
What is a pretext task?
The made-up practice task the model solves during self-supervised learning, such as filling in a hidden word or predicting a missing patch of an image. The task is a means to an end: it forces the model to learn general patterns.