ControlNet in AI: what it is, explained simply

In plain words

Words are bad at describing exact shapes. Type "a dancer with her left arm raised" into a text-to-image tool and you might get the right arm, or both.

ControlNet fixes this by adding a second input: a simple guide picture. It could be a stick figure showing the pose, the outline of a building, or a rough map of what is near and far. The words decide what things look like. The guide decides where they go.

It was published in February 2023 by researchers at Stanford and quickly became a standard tool for people using open image models such as Stable Diffusion, because it gives artists and designers real control over composition.

Everyday example

A rangoli stencil

A rangoli stencil fixes the pattern: where every petal and curve goes. You still choose the colours and powders, and two people with the same stencil can make very different rangolis. ControlNet's guide picture is the stencil, and your prompt is the choice of colours.

Where the example breaks: A stencil forces the powder to stay inside its holes. ControlNet only nudges the model strongly in that direction, and you can set how strongly. With a weak setting or an odd guide, the picture can drift from it.

How it actually works

ControlNet is not a new image generator. It is a trainable attachment to an existing diffusion model, whose original weights stay frozen.

Make a guide, often automatically

Guides are called conditions. The original release offered edges, straight lines, soft outlines, scribbles, human pose, segmentation maps, depth and surface normals. Most can be extracted from an existing photo by a detector, for example a pose estimator that turns a photo of a person into a stick figure.

Copy the model's encoder

ControlNet makes a trainable copy of the encoder half of the diffusion model's network and keeps the original locked. The copy reads the guide picture. Because it starts from the original's weights, it already understands images well.

Join them with zero convolutions

The copy's outputs are added back into the locked model through "zero convolutions": tiny layers whose weights start at exactly zero. At the start of training they add nothing, so the original model is not disturbed, and the influence grows only as it is learned.

Train on pairs, then generate

Training uses pairs of images and their extracted guides. The paper reports robust training from fewer than 50,000 pairs to more than a million. At generation time, a strength setting chooses how closely to follow the guide, and several ControlNets can be combined, say pose plus depth.

Published

February 2023, by Lvmin Zhang, Anyi Rao and Maneesh Agrawala at Stanford University; presented at ICCV 2023

Built for

Stable Diffusion, the open diffusion model; later versions exist for other open models

Original guide types

Edges, lines, soft outlines, scribbles, human pose, segmentation, depth and normal maps

Used for

Matching a pose, turning sketches or floor plans into renders, keeping a layout while changing style, product mock-ups

ControlNet fits a family of add-ons, like LoRA, that adapt a big model to a new job without retraining the whole thing.

Questions people ask

What does ControlNet do in Stable Diffusion?

It adds a guide picture, such as a pose, edge outline or depth map, so the generated image follows that structure. Your text prompt still decides content and style. Without it, Stable Diffusion picks the layout on its own.

Is ControlNet the same as image-to-image?

No. Image-to-image starts from your actual picture under noise, so colours and details carry over. ControlNet only takes the structure, like a stick figure or outline, so you can get a completely new look in exactly the same pose or layout.

Does ControlNet change the original model?

No. The original model's weights are locked. ControlNet trains a separate copy of part of the network and feeds its signal in through layers that start at zero, so the base model keeps everything it knew.

What is a ControlNet pose?

A pose condition is a stick-figure skeleton showing where a person's head, shoulders, elbows, hands, hips, knees and feet are. It is usually extracted from a reference photo with a pose detector, then used to make a new person in the same position.