In plain words
Take photos of a room or a statue from many angles. Gaussian splatting turns them into a 3D scene made of millions of tiny fuzzy blobs, each with its own position, shape, colour and see-through-ness.
Seen from far, the blobs blend into a convincing picture, the way dots of paint blend in a pointillist painting. Move the virtual camera and the computer simply flattens every blob onto the screen and layers them front to back. That flattening is the "splat".
It does the same job as a NeRF: new views of a real place from ordinary photos. The difference is speed. There is no neural network to ask at every point, so it can draw the scene in real time on a good graphics card.
Everyday example
A rangoli made of coloured powder
Up close, a rangoli is just soft dabs of powder, each a blur of one colour. Step back and the dabs merge into a sharp peacock. Now imagine the dabs floating in 3D, with some stretched long and thin for edges and some faint for glass. View them from any side and the picture forms again.
Where the example breaks: A rangoli artist decides where each dab goes. Here the computer places and reshapes the blobs by trial and error until they match the photos, with no idea what a peacock or a chair is. Spots no photo saw come out as smears.
How it actually works
The method, 3D Gaussian Splatting, was published in 2023. A "Gaussian" is a soft blob that is densest in the middle and fades out smoothly, like a puff of smoke with a clear centre.
Start from a rough point cloud
Software first works out where each photo was taken from, and as a side effect finds a sparse cloud of 3D points on the scene. Each point becomes a starting blob.
Give every blob four kinds of numbers
Each blob stores a position, a shape (how far it is stretched in each direction and how it is turned), an opacity, and a colour that can change slightly with viewing angle, so shiny surfaces look more natural.
Draw, compare, adjust
Render the blobs from one photo's viewpoint, measure the difference from the real photo, and use gradient descent to adjust every blob's numbers. It works like training a network, but the blobs themselves are what is learned.
Split, copy and prune
During training, blobs in under-covered areas are copied, oversized ones are split into smaller ones, and nearly invisible ones are deleted. A finished scene typically ends up with one to five million blobs.
Render by splatting
To show a frame, each blob is projected into a flat 2D smudge, sorted by distance and blended front to back on the GPU. The original paper reports at least 30 frames a second at full HD.
- First described
- August 2023, "3D Gaussian Splatting for Real-Time Radiance Field Rendering", presented at SIGGRAPH 2023
- Who
- Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis, at Inria, Université Côte d'Azur and the Max Planck Institute for Informatics
- Training time
- Under an hour per scene in the paper's tests, versus about 48 hours for the best NeRF-style method it compared against
- The trade-off
- Files of several hundred megabytes per scene, against roughly 10 to 50 megabytes for the NeRF-style methods it compared against
Splats are used to capture real places and objects in 3D, and as a building block in 3D generation research.
Questions people ask
Why is it called Gaussian splatting?
"Gaussian" is the bell-curve shape of each blob: dense in the middle, fading at the edges. "Splatting" comes from older volume-rendering work that spreads each 3D sample onto the screen as a flat footprint, like a splat of paint.
Is Gaussian splatting AI?
It uses the same training method as machine learning: draw, compare with real photos, adjust by gradient descent. But there is no neural network. The learned result is a big list of blobs, which is why it is fast to draw.
Gaussian splatting vs NeRF: what is the difference?
Both rebuild a real scene from photos. A NeRF stores it inside a neural network, which is compact but slow to draw. Gaussian splatting stores it as millions of blobs, which draws in real time and trains faster than the original NeRF, but takes much more storage.
Can I make a Gaussian splat with my phone?
Often, yes. You record a slow video or many photos around a still object or room. Software then estimates the camera positions and trains the blobs. Some apps do all of this on the phone itself; others use a computer or a cloud service. Anything that moves during capture blurs, because the method assumes a still scene.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change