In plain words
Many AI image and video tools, such as Stable Diffusion and DALL-E 2 and 3, are built on diffusion models.
They work backwards from how you might expect. Instead of painting a picture stroke by stroke, the model starts with a square of random static, like an old TV with no signal. Then it asks: if a real image were hidden under this noise, what would a slightly cleaner version look like? It removes a little noise, and repeats this dozens of times.
Your prompt steers every step. With "a sun over green hills", each cleanup nudges the static towards sun-like and hill-like shapes, until a clear picture is left.
Everyday example
Wiping a fogged-up mirror
After a hot shower, the bathroom mirror is completely fogged. Wipe a little and shapes appear. Wipe more and you see a face. A diffusion model "wipes" noise away in small passes. The difference: there is no real face behind the fog. The model invents what should be there, guided by your words.
Where the example breaks: A mirror hides a real reflection. Behind the model's noise there is nothing. It is generating a new image from what it learned about millions of pictures, which is also why details like hands and text sometimes come out wrong.
How it actually works
The name comes from physics: diffusion is how a drop of ink spreads through water until it's evenly mixed. The model learns to run that process in reverse.
Training: destroy images on purpose
Take millions of captioned images. For each one, add random noise in many small steps until only static remains. This forward process needs no learning, it is just maths.
Learn to predict the noise
A neural network is shown a noisy image, how noisy it is, and its caption, and trained to predict exactly which noise was added. Get that right, and you can subtract it to get a slightly cleaner image.
Generate: start from static
To make a new image, start from fresh random noise. The network predicts the noise, a little is removed, and the loop repeats. The first diffusion papers used about 1,000 steps. Modern samplers need around 20 to 50, and some distilled models just a few.
Words steer the cleanup
Your prompt is turned into an embedding by a text encoder. At every step, the network looks at it through attention, so the noise is removed in a direction that matches the description.
Work in a compressed space
Denoising millions of pixels is slow. Latent diffusion, the idea behind Stable Diffusion, first compresses the image into a much smaller latent space, denoises there, and decodes the result back into pixels at the end.
- First proposed
- 2015, by Jascha Sohl-Dickstein and colleagues at Stanford and UC Berkeley
- Breakthrough
- 2020, "Denoising Diffusion Probabilistic Models" by Jonathan Ho, Ajay Jain and Pieter Abbeel at UC Berkeley
- Went mainstream
- 2022: DALL-E 2, Google's Imagen and the open Stable Diffusion
- Beyond images
- Video (Sora and others), music, speech and even protein design
Before diffusion, the leading image generators were GANs. Diffusion models proved more stable to train and better at covering the full variety of images, so they took over.
Questions people ask
Why is it called a diffusion model?
Because training mimics diffusion in physics, like ink spreading in water until it is uniform. Images are gradually turned into uniform noise, and the model learns to reverse that spreading.
Is ChatGPT a diffusion model?
No. ChatGPT's text comes from a large language model that predicts the next word. Image generation inside chat apps has used diffusion models and, more recently, other approaches built into the language model itself.
Why do AI images get hands and text wrong?
Hands come in many poses and are often partly hidden in photos, so the patterns are hard to learn. Text needs exact letter shapes, while diffusion works with rough shapes first. Newer models handle both much better.
What is the difference between a diffusion model and a GAN?
A GAN makes an image in one go, trained by a generator competing with a critic. A diffusion model builds an image over many small denoising steps. Diffusion is slower to run but steadier to train and more varied.
Does a diffusion model copy images it was trained on?
Usually it generates new combinations, but research has shown that models can sometimes reproduce near-copies of images that appeared many times in training. This is part of the debate around AI and copyright.
Further reading
- Denoising Diffusion Probabilistic Models (Ho, Jain, Abbeel)arXiv, 2020
- Deep Unsupervised Learning using Nonequilibrium Thermodynamics (Sohl-Dickstein et al.)arXiv, 2015
- High-Resolution Image Synthesis with Latent Diffusion Models (Rombach et al.)arXiv, 2021
- Extracting Training Data from Diffusion Models (Carlini et al.)arXiv, 2023