In plain words
A photo shows a moment. A video shows what changes: a ball being bowled, a cup tipping over, someone walking in and sitting down. Video understanding is about following that change, not just naming what is on screen.
Most systems look at still frames sampled from the video, often about one per second, plus the sound or a transcript. A vision language model turns each frame into tokens and reads them in order, the way it reads words in a sentence.
That lets you ask things like "When does the speaker show the chart?" or "Did anyone leave the bag behind?" and get an answer with a timestamp. The catch: anything that happens between the sampled frames can be missed, and long videos use up a lot of the model's context window.
Everyday example
Catching up on a match from highlights
You missed the match, so a friend sends you one photo from every over. Flip through them and you can still tell the story: who was batting, when the score jumped, when a wicket fell. But a quick run-out between two photos? You would never know it happened. A video model works from samples like these.
Where the example breaks: Your friend chose the most important moments. A model usually takes frames at fixed intervals, whether or not anything interesting is happening, so it can waste effort on dull stretches and skip the one second that mattered.
How it actually works
Video adds a time axis to computer vision. The hard parts are motion and length: what moved, in what order, across minutes or hours.
Early models: appearance plus motion
In 2014, a "two-stream" network used one branch to look at still frames and another to look at optical flow, a map of how each pixel moved between frames. Together they recognised human actions in short clips.
Big labelled video datasets
DeepMind's Kinetics dataset (2017) collected 400 human actions with at least 400 clips each, roughly ten seconds long, from YouTube. Its classes range from making tea to playing the violin.
Transformers cut video into space-time patches
Video transformers, such as ViViT (2021), split a clip into small patches across both space and time, turn each into a token, and use attention to relate them. Movement becomes relationships between tokens in different frames.
Language models that watch
Today's multimodal models sample frames, encode them as tokens, add the audio or a transcript, and let you ask questions in plain language. As of November 2026, Google's Gemini API samples one frame per second by default and notes fast action can lose detail.
Long videos are the frontier
An hour of video can be hundreds of thousands of tokens. Benchmarks such as Video-MME (2024), with clips from 11 seconds to an hour, found models still struggle more as videos get longer.
- Two-stream networks
- 2014, Karen Simonyan and Andrew Zisserman, University of Oxford
- Kinetics dataset
- 2017, DeepMind: 400 action classes, about 10 seconds per clip
- Typical sampling
- About 1 frame per second in many chat models, adjustable in some
- Where you meet it
- Asking questions about a clip, finding the moment something happens, summarising talks and lessons, describing what is on screen
Following one object through a video is a narrower task called object tracking. Video understanding is the broader job of knowing what is going on.
Questions people ask
Can AI chatbots watch videos?
Some can take a video file or link and answer questions about it. Under the hood they usually sample frames and read the audio, rather than looking at every frame. Google's Gemini API, for example, documents one frame per second plus a low-quality audio track. Support differs by app and changes often.
How does AI summarise a video?
It can draw on two things: what is said, from the audio or a transcript, and what is shown, from sampled frames. Using both, a model can mention the chart on screen, not just the words spoken over it.
Why does AI miss things in videos?
Because it usually looks at a sample of frames, not all of them. A video at 30 frames a second sampled once a second shows the model one frame in thirty. Quick actions, small details and text on screen can fall through the gaps.
Is video understanding the same as text-to-video?
No. Video understanding reads a video and describes it. Text-to-video goes the other way: it makes a new video from a description.
Was this clear?
1 is confusing, 5 is crystal clear. One tap, no account needed.
Spotted something wrong, or know a better example? Suggest a change
Further reading
- Video understandingGoogle AI for Developers (Gemini API)
- Two-Stream Convolutional Networks for Action Recognition in Videos (Simonyan, Zisserman)arXiv, 2014
- The Kinetics Human Action Video Dataset (Kay et al.)arXiv, 2017
- ViViT: A Video Vision Transformer (Arnab et al.)arXiv, 2021
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (Fu et al.)arXiv, 2024