In plain words
Take a photo of your street and a painting you love, say one full of swirling blue brushstrokes. Style transfer makes a new picture that shows your street, painted the way that artist paints.
It works because an image AI can pull two things apart. One is the content: what is in the picture and where it sits, like the houses, the tree, the road. The other is the style: the colours, textures and kinds of strokes, wherever they appear.
The AI keeps the content of the first picture and copies the style of the second. That is the idea behind many "turn my photo into a painting" filters. Newer image-to-image tools can also restyle a photo from a written request, using a different technique.
Everyday example
Singing a film song in a qawwali style
A singer takes a Bollywood song everyone knows and performs it as a qawwali. The words and the tune stay the same, so you still recognise the song. What changes is the clapping, the harmonium, the way the voice rises. Style transfer does this to pictures: same scene, new way of painting it.
Where the example breaks: A singer understands both the song and the qawwali tradition. The AI understands neither. It only matches statistics of colour and texture it measures inside a network, so it can smear faces or put brushstrokes where a painter never would.
How it actually works
Neural style transfer was introduced in 2015. It uses a convolutional neural network already trained to recognise objects, and reads content and style from different layers of it.
Use a trained image network as a measuring tool
The original method used VGG, a 19-layer network trained for image recognition. Nobody retrains it. Its layers simply turn any picture into feature maps: grids saying where it found edges, textures, parts and objects.
Content comes from the deeper layers
Deeper layers respond to what objects are and where they sit, and care less about exact pixels. So the content of your photo is described by those layers' responses.
Style comes from how features co-occur
For style, the method measures which features tend to appear together, across the whole image, in several layers. These correlations (a Gram matrix) capture colours and textures but throw away where things are. That is why style can move to a new scene.
Nudge a new image until both match
Start from random noise and adjust the pixels step by step with gradient descent until the image's content matches the photo and its style matches the painting. A dial weighs the two: more content or more style.
Make it fast
That search took many steps for each picture. In 2016, researchers trained a network to do it in one pass, about a thousand times faster, but one network per style. In 2017, a method called AdaIN handled any new style in real time.
- First described
- August 2015, "A Neural Algorithm of Artistic Style" by Leon Gatys, Alexander Ecker and Matthias Bethge
- Famous demo
- A photo of riverside houses in Tübingen, repainted as van Gogh's The Starry Night, Munch's The Scream and works by Turner, Picasso and Kandinsky
- Speed up
- 2016, feed-forward networks about three orders of magnitude faster than the original search
- Where you meet it
- Photo and video filters that make pictures look painted, sketched or cartoon-like
Style transfer separated "what" from "how it looks" years before today's diffusion models. Those models can restyle too, but by redrawing the picture from a prompt rather than matching feature statistics.
Questions people ask
How do apps turn photos into paintings?
Many use style transfer or a newer image-to-image model. Both keep the layout of your photo and change its colours, textures and strokes to match a chosen look. The fast versions run a trained network once, so the result appears in seconds.
Is style transfer the same as a filter?
A classic filter applies the same fixed rule to every pixel, like warming the colours. Style transfer uses a neural network that looks at the whole picture, so it can add new brushstrokes and textures that were never in your photo.
Can style transfer copy any artist?
Technically it can borrow the look of almost any image you give it. Whether you should is another matter: copying a living artist's style raises fair questions, and the law on AI and art is still being argued. See AI and copyright.
Does style transfer work on video?
Yes, but it is harder. Restyling each frame on its own makes the brushstrokes flicker. Video methods add a check that neighbouring frames stay consistent, so the painted look holds still as things move.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change
Further reading
- A Neural Algorithm of Artistic Style (Gatys, Ecker, Bethge)arXiv, 2015
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution (Johnson, Alahi, Fei-Fei)arXiv, 2016
- Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization (Huang, Belongie)arXiv, 2017
- Neural style transferTensorFlow