AppsGuidesAI DictionarySmall FixesOn RecordAboutFollow the lab

Model serving

Also called inference server, vllm

Model serving is the software and machines that keep a trained AI model running, take in requests from many people, and send each answer back quickly.

Chips and scaleTechnical

In plain words

Training makes a model. Serving puts it to work. When you send a message to a chatbot, it lands on a server that already has the model loaded into GPU memory. Serving software queues your request, runs the model, and streams the answer back piece by piece.

The hard part is doing this for many people at once. GPUs are expensive, and serving one person at a time would leave most of the chip idle. So serving systems group many people's requests together and run them through the model in one go, while keeping each person's wait short.

Popular open-source serving engines include vLLM, first built at UC Berkeley, and SGLang. Smarter serving software is one reason the same chips can answer more people for less money, which helps push down inference costs.

Everyday example

A dosa counter at lunch rush

One cook, one huge tawa. A poor counter makes each customer wait until the previous dosa is done. A good one spreads batter for several orders at once, slides a new order onto any free spot the moment a dosa comes off, and keeps a token system so nobody is forgotten. Model serving is the good counter: one costly GPU, many requests, none left waiting long.

Where the example breaks: Dosas do not need to remember anything. Each AI request keeps its own memory of the conversation so far, called the KV cache, and running out of room for that memory is what limits how many fit at once.

Go deeper

How it actually works

A serving stack sits between users and the model. Its job is to keep high throughput and low waiting time at the same time.

  1. Load the model and open a door

    The server loads the model's parameters into GPU memory once and keeps them there. It offers an API, often in the same format as OpenAI's, so apps can call it like any cloud model.

  2. Continuous batching

    Older systems grouped requests and waited for the slowest one to finish. The Orca paper (2022) schedules work at every step of generation instead: finished requests leave and new ones join between every token. On a 175-billion-parameter model it reported up to 36.9 times more throughput at the same delay.

  3. Managing memory

    Each request keeps a KV cache that grows as it writes. vLLM's PagedAttention (2023) stores it in small fixed-size pages, like a computer's virtual memory, so little memory is wasted and more requests fit. Its paper reported 2 to 4 times more throughput.

  4. Splitting prefill and decode

    Reading your prompt (prefill) is one big burst of maths. Writing the answer (decode) is a long series of small steps. Running both on the same GPUs makes them get in each other's way, so some systems put them on separate GPUs. The DistServe paper (2024) reported up to 7.4 times more requests served within delay targets.

  5. Extra speed tricks

    Serving engines also use quantization (storing numbers in fewer bits), speculative decoding and prefix caching, and spread very large models across several GPUs.

Continuous batching
Orca, OSDI 2022, by researchers at Seoul National University and FriendliAI
vLLM
Built at UC Berkeley (PagedAttention paper, 2023); a PyTorch Foundation project since May 2025
Key measures
Time to first token, time per output token, and total throughput across users (see latency and tokens per second)
As of November 2026
Hugging Face's TGI server is in maintenance mode; Hugging Face recommends vLLM, SGLang, llama.cpp or MLX instead

Questions people ask

What is the difference between inference and model serving?

Inference is the act of running a model on an input to get an answer. Model serving is the whole system around it: loading the model, accepting requests, batching them, managing memory, and sending answers back to many users reliably.

What is vLLM?

vLLM is a free, open-source engine for serving large language models. It began at UC Berkeley around a memory technique called PagedAttention, published in 2023, and is now hosted by the PyTorch Foundation. It runs on Nvidia and AMD GPUs, ordinary processors and other chips.

What is continuous batching?

A way of sharing a GPU between many requests. Instead of waiting for a whole group to finish, the server adds new requests the moment others finish, at every step of writing. The GPU stays busy and nobody waits for the slowest request.

Do I need a serving system to run a model on my laptop?

Not a big one. For one person, a local engine such as llama.cpp or MLX is enough. Serving systems matter when many people share the same hardware and every second of GPU time costs money.

Was this clear?

1 is confusing, 5 is crystal clear. One tap, no account needed.

Spotted something wrong, or know a better example? Suggest a change

Further reading

Prepared by Mundhra Labs. Reviewed by Aditya Mundhra. Checked November 2026.