In plain words
Most AI needs labelled examples, but labels are costly. Contrastive learning finds another teacher: comparison. Show a model two things that belong together and several that do not, and ask it to tell which is which.
A common way to make matching pairs without any human help is to take one photo and make two edited copies, say one cropped and one with changed colours. They are still the same picture, so they should match. Every other photo in the batch counts as a mismatch.
The model turns each input into a list of numbers, an embedding, like a spot on a map. Training pulls the spots of matching pairs together and pushes mismatches apart. To win this game, the model has to notice what really makes a picture what it is, ignoring the crop and the colours. It is a form of self-supervised learning, and the same idea taught CLIP which captions match which images.
Everyday example
Spotting your friend across photos
Someone shows you a wedding photo of your friend in a sherwani, a beach photo of the same friend in shorts, and ten photos of strangers. You must pick the two that show the same person. To get it right every time, you learn to ignore the clothes and the background and look at the face. Contrastive learning trains a model through thousands of rounds of this game.
Where the example breaks: You know your friend. The model is told which pairs match only by how they were made, so two different dog photos count as a mismatch even though both show a dog, and it learns only what the chosen edits make it ignore.
How it actually works
Contrastive learning trains an encoder so that distance in its embedding space reflects similarity.
Make positive pairs
A positive pair is two inputs that should match: two augmented views of one image (random crop, colour shift, blur), or an image and its caption. The choice of edits decides what the model learns to ignore. SimCLR found that combining several edits is critical.
Use the rest as negatives
Within a training batch, every other example is treated as a negative for a given input. More negatives make the task harder and more informative, which is why contrastive methods benefit from large batches and long training.
Score with a contrastive loss
Both views pass through the same neural network to become embeddings. A loss function, such as the one SimCLR calls NT-Xent, compares the similarity of the true pair against all the negatives, a bit like a classification task where the right answer is the matching partner.
Reuse the learned features
After training, the encoder's embeddings are reused: a small classifier trained on top, with few labels, can do well. SimCLR features with a simple linear classifier on top reached 76.5% top-1 accuracy on ImageNet, matching a standard supervised ResNet-50, a widely used image model, although SimCLR needed a network four times wider to do it.
Across text and images
CLIP applies the same idea across two kinds of data. Trained on 400 million image and caption pairs from the internet, it learned to score which caption goes with which image, and its text encoder was later reused inside text-to-image systems such as the original Stable Diffusion.
- Early use of the name
- A "contrastive loss" in Hadsell, Chopra and LeCun's 2006 work at New York University
- Landmark image method
- SimCLR, by Ting Chen and colleagues at Google, 2020
- Landmark image and text method
- CLIP, OpenAI, 2021
- Used for
- Pre-training image encoders that need few labels later, and matching images with text
Questions people ask
What are positive and negative pairs?
A positive pair is two inputs that should be treated as the same, such as two edited copies of one photo. A negative pair is two inputs that should be told apart, usually two different examples from the same batch.
Is contrastive learning supervised or unsupervised?
It is usually self-supervised: the pairs are built from the data itself, with no human labels. The labels it later helps with, such as naming objects in photos, come from a much smaller labelled set used afterwards.
How is CLIP related to contrastive learning?
CLIP is trained contrastively. In each batch, every image's true caption is the positive and all the other captions are negatives, so the model learns a shared space where pictures and matching words sit close together.
Why does contrastive learning need large batches?
Each example uses the rest of its batch as negatives. A bigger batch means more and harder comparisons, which gives a stronger learning signal. SimCLR reported that it benefits from larger batches and longer training than supervised learning does.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change
Further reading
- A Simple Framework for Contrastive Learning of Visual Representations (Chen et al.)arXiv, 2020
- Advancing Self-Supervised and Semi-Supervised Learning with SimCLRGoogle Research, 2020
- Learning Transferable Visual Models From Natural Language Supervision (Radford et al.)arXiv, 2021
- Dimensionality Reduction by Learning an Invariant Mapping (Hadsell, Chopra and LeCun)CVPR / New York University, 2006