Image segmentation: what it is, explained simply

In plain words

Ordinary object detection draws a rough box around a car and says "car". Image segmentation goes further. It decides, for every single pixel, whether it is car, road, person, tree or sky.

The result is like a colouring book filled in perfectly: each object gets its exact shape, edges and all. That precision is what lets your phone blur the background behind your face in portrait mode, or lets you lift a pet out of a photo with one tap.

Doctors use it to outline a tumour on a scan. Self-driving cars use it to know exactly where the road ends and the footpath begins. Photo editors use it to select things without tracing them by hand.

Everyday example

Sorting a thali by hand

Look at a full thali. A box-drawing AI would say "dal somewhere here, rice somewhere there". Segmentation is like tracing exactly where the dal ends and the rice begins, grain by grain, and colouring each item differently. Two rotis get two separate outlines, not one blob.

Where the example breaks: You know what dal is and why it matters. A segmentation model only learns which pixel patterns usually belong together. At blurry edges, like hair against a busy background, it often guesses, which is why cut-outs sometimes chop off strands.

How it actually works

Segmentation is one of the core jobs of computer vision. It comes in three flavours.

Semantic: a label for every pixel

Every pixel gets a class such as road, person or sky. Two people standing together become one "person" region. That is enough for jobs like telling road from footpath.

Instance: each object separately

Each object gets its own mask, so person 1 and person 2 are told apart. Panoptic segmentation, named in a 2018 paper, combines both: every pixel labelled, and every countable object kept separate.

How the network does it

A convolutional network or vision transformer first shrinks the image into rich features, then a decoder blows them back up to full size and scores each pixel. U-Net, a 2015 design for microscope images, made this shrink-then-expand shape famous and trained well from few labelled images.

Promptable: segment anything

Meta's Segment Anything Model (2023) can cut out almost any object you click on, even kinds it never saw labelled. It was trained on over 1 billion masks across 11 million images. SAM 2 (2024) extended this to video, and SAM 3 (November 2025) segments every match for a short phrase like "yellow school bus".

Classic design

U-Net, May 2015, University of Freiburg: segmented a 512 by 512 image in under a second on a GPU

Big dataset

SA-1B, April 2023: over 1 billion masks on 11 million licensed images, released by Meta

Newest Segment Anything, as of October 2026

SAM 3, released November 2025, with a SAM 3.1 update in March 2026

Uses

Portrait mode, background removal, medical scans, self-driving cars, photo editing

Segmentation often does the selecting for other tools: inpainting needs a mask, and object tracking can follow a mask from frame to frame.

Questions people ask

What is the difference between object detection and image segmentation?

Object detection draws a rectangle around each object and names it. Segmentation outlines the exact shape, pixel by pixel. Detection is faster and enough for counting cars. Segmentation is needed when the precise edge matters, like cutting out a person or measuring a tumour.

What is Segment Anything (SAM)?

A family of open models from Meta that cut out objects when you click, draw a box or, since SAM 3, type a short phrase. The first came in April 2023. SAM 2 added video in July 2024, and SAM 3 followed in November 2025.

How does portrait mode blur the background?

The phone runs a segmentation network that estimates which pixels are person and which are background, often combined with a depth estimate. It keeps the person sharp and blurs the rest. Errors at the mask's edge are why hair or glasses sometimes look smudged.

What is semantic segmentation?

Semantic segmentation gives every pixel a category, such as road, car or sky, without telling separate objects of the same kind apart. Instance segmentation does tell them apart, and panoptic segmentation does both at once.