In plain words
Older image AI learned a fixed list of labels, like 1,000 kinds of objects. Ask it about anything else and it was stuck.
CLIP learned differently. It was shown 400 million pictures from the internet with the text that came with them. Its only job was a matching game: given a picture, pick its real caption out of a big pile of wrong ones.
To win that game, it had to learn what words look like in pictures. So now you can give it any image and any words, like "a cat wearing sunglasses" or "a crowded railway station", and it tells you how well they match. No new training needed.
That one ability powers search-your-photos-by-words tools and parts of many text-to-image systems.
Everyday example
Matching wedding photos to their captions
A relative hands you a stack of wedding photos and a separate list of captions: "mehendi", "baraat", "pheras", "vidaai". You match each photo to its caption. After doing this for thousands of weddings, you could caption photos from a wedding you have never seen. CLIP played that game 400 million times.
Where the example breaks: You know what a baraat is and why it happens. CLIP only learned which patterns of pixels tend to appear with which words online, including the internet's mistakes and stereotypes.
How it actually works
CLIP stands for Contrastive Language-Image Pre-training. It is two encoders trained together so that pictures and text land in one shared embedding space.
Two encoders, one space
An image encoder (a CNN or vision transformer) turns a picture into a list of numbers, a vector. A text encoder turns a caption into a vector of the same length. If picture and words mean the same thing, their vectors should point the same way.
Contrastive training
In each training batch, CLIP sees many pictures and their captions. It is rewarded when each picture's vector is closest to its own caption's vector and pushed away from every other caption. In the original blog, each picture had to find its caption among 32,768 text snippets.
Zero-shot classification
To sort photos into new categories, write each category as text, such as "a photo of a dog", and pick the closest match. With no ImageNet training images, CLIP matched the accuracy of the original ResNet-50, a model trained on all 1.28 million of them.
A building block for other models
CLIP's encoders are reused widely. The original Stable Diffusion used CLIP's text encoder to turn prompts into vectors, DALL-E 2 was built around CLIP's image vectors, and the LLaVA vision-language model connected CLIP's image encoder to a language model.
- Released
- January 2021 by OpenAI, with the paper and model weights made public
- Training data
- 400 million image and text pairs collected from the internet
- Known weaknesses
- Counting objects and other abstract tasks; OpenAI also reported it reflected harmful social biases in tests
- Open versions
- OpenCLIP, a community project, trained CLIP-style models on public datasets such as LAION-2B
CLIP itself never makes pictures or writes captions. It only scores how well a picture and some words go together.
Questions people ask
What does CLIP stand for in AI?
Contrastive Language-Image Pre-training. Contrastive means it learns by comparing right pairs against wrong ones. Language-image means it connects text and pictures. Pre-training means this general skill is learned first, before any specific use.
Does CLIP generate images?
No. CLIP only measures how well an image and a piece of text match. Image generators use it as a helper: its text encoder can turn your prompt into numbers the generator understands, or its scores can check whether a result fits the prompt.
What is zero-shot classification?
Sorting things into categories the model was never specifically trained on. With CLIP you just write the category names as text, and it picks the one whose words best match the picture. No labelled examples of those categories are needed.
Is CLIP biased?
It can be. It learned from internet pictures and their captions, which carry stereotypes. OpenAI's own tests found it could put photos of people into offensive categories when such labels were offered, which is one reason careful testing matters before using it. See AI bias.