In plain words
A video is just a quick run of still pictures, often 30 a second. Object detection can find every car in each picture. But it does not know that the red car in this frame is the same red car as in the last one.
Object tracking adds that memory. It gives each object an ID, like "car 7", and keeps the same number on it as it moves. If a bus briefly hides it, a good tracker guesses where it went and picks it up again on the other side.
This is how traffic cameras count vehicles without counting one twice, how sports broadcasts follow a ball or player, and how a phone camera keeps focus locked on a moving child.
Everyday example
Keeping an eye on your bag at a busy station
At a crowded railway platform, you glance away and back at your bag every few seconds. Each glance is a frame. You match what you see to where the bag was and where it might have been moved. If a porter blocks your view, you look for it where it should reappear. A tracker does this for every object.
Where the example breaks: You recognise your bag by its scratches and the tag your mother tied on. A tracker only matches position, motion and a summary of appearance, so two similar objects crossing paths can swap IDs.
How it actually works
Most trackers follow a recipe called tracking by detection: find objects in every frame, then link them across frames.
Detect in every frame
A detector, usually a convolutional network or vision transformer, draws boxes around the objects in each frame. The SORT paper (2016) found that the quality of these detections is a key factor in how well tracking works.
Predict where each object goes next
For every object already being tracked, a simple motion model, often a Kalman filter, predicts where its box should be in the next frame from its speed and direction so far.
Match predictions to new detections
The tracker pairs each prediction with the closest new box, using an assignment method such as the Hungarian algorithm. Matched objects keep their ID. Unmatched boxes start new tracks, and tracks with no match for a while are dropped.
Use appearance to survive hiding
DeepSORT (2017) added a learned description of what each object looks like, so it could be re-found after being hidden. That cut ID switches, where two objects swap labels, by 45%. Newer models such as Meta's SAM 2 (2024) follow an object's exact outline using a memory of earlier frames.
- Classic recipe
- SORT, February 2016: Kalman filter plus Hungarian algorithm, updating 260 times a second
- Appearance added
- DeepSORT, March 2017: 45% fewer identity switches
- In cricket
- Hawk-Eye ball tracking was first used on TV in 2001, combining up to ten high-speed cameras into a 3D path
- Uses
- Traffic counting, sports analysis, camera autofocus, warehouse robots, self-driving cars, wildlife studies
Tracking people raises the same questions as facial recognition: who is being followed, by whom, and with what consent.
Questions people ask
What is the difference between object detection and object tracking?
Detection finds objects in a single picture. Tracking links those detections across a video so each object keeps the same ID over time. You need tracking to count things that move, measure speed or follow a path.
What is an ID switch?
An ID switch happens when a tracker mixes up two objects, for example when two people cross and their labels swap. It is one of the main errors trackers are judged on, because it breaks counts and paths.
How does Hawk-Eye track the ball in cricket?
Several high-speed cameras around the ground film the ball from different angles. Software finds the ball in each camera's frames and combines the views to work out its path in 3D, which is then used for replays and LBW reviews.
Can object tracking identify who a person is?
Not by itself. A tracker knows "person 4 walked from here to there", not their name. Linking a track to an identity needs something extra, such as facial recognition, which is why the two together are closely regulated in some places.