Black October, explainedAppsGuidesAI DictionarySmall FixesOn RecordAboutFollow the lab

Vision-language model

Also called VLM, vision model

A vision-language model is AI that can look at an image and talk about it: describe it, read it, or answer your questions about it.

Images, voice and videoTechnical

In plain words

Send a chatbot a photo of your lunch and ask "how many rotis?", or a screenshot of an error and ask "what went wrong?", and it answers. The model behind that is a vision-language model, or VLM.

It works by turning the picture into the same kind of input a language model already reads. The image is cut into small square pieces, and each piece becomes a token, a little bundle of numbers, just like a word does. The model then reads the picture pieces and your words together, in one sequence, and writes a reply.

This is how today's chatbots read charts, receipts, handwriting and diagrams. It is one part of Multimodal AI, AI that handles more than one kind of input. It is powerful, but it can still miscount objects or invent details that are not in the picture, a visual kind of hallucination.

Everyday example

Describing a photo over the phone

A friend holds up their phone at a shop and asks you, on a video call, "Which of these two kurtas matches the dupatta?" You look at the picture and their question together, then answer in words. A vision-language model does the same: it takes in the image and the question as one thing and replies.

Where the example breaks: You see the colours and know what a dupatta is. The model turns the image into numbers and links them to patterns it learned from captioned pictures. It has never seen a real kurta, so lighting tricks or unusual items can mislead it.

Go deeper

How it actually works

Most VLMs bolt a vision encoder onto a large language model.

  1. Cut the image into patches

    A vision encoder, usually a vision transformer (ViT), splits the image into a grid of small patches. The 2020 ViT paper used patches of 16 by 16 pixels and showed that a plain transformer could classify images this way.

  2. Learn what pictures mean

    The encoder is often pre-trained to match images with captions. OpenAI's CLIP (2021) learned this from 400 million image and text pairs collected from the internet, placing pictures and their descriptions close together in a shared embedding space.

  3. Translate into the language model's tokens

    A small connector, called a projector, turns each patch's vector into the same format as word tokens. The LLM then sees picture tokens followed by text tokens, and uses attention to relate them: "plate" can attend to the patches showing the plate.

  4. Tune it to follow instructions

    Finally the combined model is instruction-tuned on examples of images with questions and good answers, so it can chat about pictures rather than only caption them.

Early landmark
April 2022: DeepMind's Flamingo answered questions about images and video from just a few examples
Open recipe
April 2023: LLaVA showed a simple recipe: connect a vision encoder to an LLM, then tune it on visual instructions
Mainstream
September 2023: OpenAI rolled out image understanding in GPT-4 (GPT-4V). As of October 2026, Claude and Gemini, among others, accept images
Common uses
Reading documents and charts, describing images for blind and low-vision users, visual Q&A, checking screenshots

Questions people ask

What is a VLM in AI?

VLM stands for vision-language model: a model that takes images and text as input and produces text. You can ask it to describe a photo, read a document, or answer a question about a chart.

What is the difference between a VLM and an LLM?

An LLM reads and writes text only. A VLM adds a vision encoder so it can also take images as input. Many VLMs are built by attaching that encoder to an existing LLM.

Is a vision-language model the same as multimodal AI?

A VLM is one kind of multimodal AI, covering images and text. Multimodal is the broader term and can also include audio and video, in and out.

Can vision-language models read handwriting and documents?

Often yes, and they can do more than OCR, such as summarising a form or answering questions about a table. They can also misread small text or invent values, so check important numbers.

Can a VLM create images?

A classic VLM only reads images and writes text. Making pictures is text-to-image. Some newer models do both inside one system, which blurs the line.

Further reading

Prepared by Mundhra Labs. Reviewed by Aditya Mundhra. Checked October 2026.