In plain words
Every AI model has two lives. First it is trained, which is slow, costly and done once. Then it is put to work answering real requests. That second part is inference.
Each time you ask a chatbot something, open your phone with your face or translate a sign, a model runs inference: your input goes in, the model does its arithmetic with the numbers it learned, and an answer comes out.
During inference the model does not learn. Its parameters stay frozen. It applies what it already knows, the way a doctor applies years of study to a new patient.
Inference happens billions of times a day around the world, so over a model's life it can cost more than training did. That is why companies work hard to make models faster and smaller.
Everyday example
A tabla player's concert
For years a tabla student practises with her guru: that is training. Then comes the concert. Someone calls out a taal and she plays it on the spot, drawing on everything she practised. She is not learning during the show, she is using what she learned, again and again, for each new request. That performance is inference.
Where the example breaks: A musician still improves a little with every concert. A model's numbers do not change at all during inference unless its makers train a new version.
How it actually works
Inference is a forward pass: data flows through the model once, from input to output, with no weight updates.
Input becomes numbers
Text is split into tokens, images into pixel values, sound into wave samples. Everything becomes numbers the model can process.
One forward pass
The numbers flow through the model's layers and are multiplied and added with its fixed weights. A classifier ends with a score for each label, and the highest wins.
Chatbots repeat it
A language model writes one token at a time. Each new token needs another forward pass, which is why long answers take longer. This loop is next-token prediction.
Where it runs
Large models run on GPUs in data centres. Smaller ones run directly on phones and laptops, called on-device AI, which can be quicker and keeps data local.
Making it cheaper
Tricks such as quantization (storing weights with fewer digits) and distillation (training a small model to copy a big one) cut the cost of every answer.
- IBM's estimate
- Up to 90% of an AI model's life is spent in inference (IBM Research, 2023)
- Also called
- Running, serving or deploying a model; "prediction" in classic machine learning
- Measured by
- Speed, such as tokens per second, and the wait before the first word
- Not the same as
- "Inference" in logic or statistics, meaning drawing conclusions from evidence
Questions people ask
What is the difference between training and inference?
Training adjusts a model's numbers using many examples, so it learns. Inference uses the finished model on a new input, so it answers. Think of a cook: training is the apprenticeship, inference is cooking each order as it comes in.
Does the model learn during inference?
No. In normal use the weights are fixed. It can use information you give it within the conversation, held in its context window, but that is gone when the chat ends unless the app saves it as memory.
Why is inference expensive?
Each answer means billions of multiplications, repeated for every token, for millions of users. Training is a big one-time bill. Inference is a meter that keeps running for as long as people use the model.
Is inference the same as reasoning?
No. In AI, inference just means running a model to get an output. Reasoning models spend extra inference time working through steps before answering, which is why people talk about inference-time compute.