In plain words
Stretch a small photo to fill a big screen and it turns blocky and soft. Old ways of enlarging just blend nearby pixels, so you get a bigger blur.
An AI upscaler has studied a great many pairs of pictures: a sharp one, and the same one shrunk and blurred. From those pairs it learned what blurry edges, hair, bricks or leaves usually look like up close. Given your small picture, it paints in that kind of detail. The proper name is super-resolution.
It is used to restore old family photos, sharpen video on smart TVs, and help games run faster by drawing frames small and enlarging them. The catch: the new detail is a confident guess, not information the camera ever recorded.
Everyday example
Reading a blurry shop sign from an auto
From a moving auto you glimpse a blurry sign: "S_ree Ganesh Sw__ts". Your brain fills it in as "Shree Ganesh Sweets" because you have seen thousands of such signs. An upscaler does the same with pixels: it fills in what similar pictures usually contain.
Where the example breaks: If the sign really said something unusual, your brain would still fill in the common version, and so would the AI. It cannot recover what the blur destroyed. It only paints what is likely.
How it actually works
Super-resolution is a classic problem in computer vision. Many different sharp images shrink to the same small one, so there is never a single correct answer, only plausible ones.
Make training pairs by damaging pictures
Take sharp, high-resolution photos and shrink, blur, add noise and compress them on purpose. Each damaged copy and its original form a pair. The 2021 Real-ESRGAN method trained only on such synthetic damage and still worked on real-world photos.
Learn the mapping from small to sharp
A convolutional network learns to turn the small image into the big one. SRCNN, published in late 2014, did this end to end. Early networks were trained to match pixels exactly, which gave safe but slightly smooth results.
Add a critic for realistic texture
GAN-based upscalers such as ESRGAN (2018) add a second network that judges whether the result looks like a real photo. This pushes the upscaler to draw crisp textures, at the risk of odd invented patterns.
Or generate the detail with diffusion
Google's SR3 (2021) used a diffusion model: start from noise and clean it up, guided by the small image. In a human test on faces enlarged from 16 by 16 to 128 by 128 pixels, people picked SR3's face over the real one close to 50% of the time, which is chance.
- Deep learning arrives
- SRCNN, submitted December 2014, by researchers at the Chinese University of Hong Kong and Microsoft Research Asia
- Realistic textures
- ESRGAN, 2018, won the PIRM2018 super-resolution challenge
- Typical enlargement
- 2 to 4 times per side in one pass. Doubling each side means four times as many pixels to fill
- Where you meet it
- Photo apps, smart TVs, video calls, game graphics such as NVIDIA's DLSS, restoring old films and photos
A 2020 face upscaler called PULSE made the limit plain: it searches for any realistic face that shrinks to your pixels, and many very different faces do.
Questions people ask
Can AI really enhance a blurry photo like in films?
Only partly. AI can make a blurry photo look sharp and believable, but the added detail is guessed from similar pictures. It cannot reveal a number plate or face that the original pixels do not contain. That is why upscaled images should never be treated as evidence.
What is the difference between upscaling and resizing?
Ordinary resizing makes each pixel bigger or blends neighbours, so the picture gets larger but no sharper. AI upscaling also enlarges, but then draws in edges and texture it has learned, so the result looks sharp.
Is AI upscaling the same as super-resolution?
Yes, for everyday purposes. Super-resolution is the research name for building a higher-resolution image from a lower-resolution one. AI upscaler is the name tools and TV makers use for the same idea.
Why do upscaled faces sometimes look like a different person?
Because a small face holds very little information, the AI fills in features from the faces it learned from. It can change eye shape, skin texture or age, and it can be biased towards the kinds of faces most common in its training data.
Further reading
- Image Super-Resolution Using Deep Convolutional Networks (Dong, Loy, He, Tang)arXiv, 2014
- High Fidelity Image Generation Using Diffusion Models (SR3)Google Research, 2021
- ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks (Wang et al.)arXiv, 2018
- Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data (Wang et al.)arXiv, 2021
- PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models (Menon et al.)arXiv, 2020