Black October, explainedAppsGuidesAI DictionarySmall FixesOn RecordAboutFollow the lab

KV cache

Also called key-value cache, KV caching

A KV cache is a chatbot's saved notes on every word it has already read, so it does not redo that work each time it writes the next word.

Inside language modelsTechnical

In plain words

A chatbot writes its answer one word at a time. Before each new word, it has to look back over everything so far: your question and every word it has already written.

Done naively, that means re-reading the whole conversation from scratch for every single word. A 500 word answer would mean 500 full re-reads.

The KV cache avoids this. The first time the model reads a word, it writes two small notes about it and keeps them. For the next word, it only makes notes for the newest word and reuses all the old ones.

The catch is memory. The notes pile up with every word, so long chats need a lot of it. This is one reason very long conversations are harder and costlier to run.

Everyday example

A shopkeeper's running bill

At a kirana store, the shopkeeper adds each item to a running bill as we put it on the counter. When we add one more packet of biscuits, he does not re-add the rice, dal and oil. He adds one line to the total he already has. The KV cache is that running bill: the past work is kept, and each new item costs only one more line.

Where the example breaks: A bill keeps one running total. The KV cache keeps a separate pair of notes for every single word, in every layer of the model, so it grows much faster than a bill and can fill a computer chip's memory.

Go deeper

How it actually works

The name comes from the keys and values inside attention, the step where each word looks back at earlier words.

  1. Attention needs keys and values for every past word

    In each layer of a transformer, every token makes a query, a key and a value. To write the next token, its query is compared with the keys of all earlier tokens, and their values are blended in.

  2. Past keys and values never change

    In chatbot models, a token can only look at tokens before it. So once a token's key and value are computed, later tokens cannot change them. That makes them safe to store and reuse.

  3. Only the newest token is computed

    With a cache, each step computes the key and value for the newest token only, adds them to the stored pile, and compares the new query against the pile. Hugging Face's documentation puts it simply: without caching, every step recomputes all previous keys and values.

  4. Memory grows with every token

    The cache holds one key and one value per token, per layer. For Meta's Llama 3.1 8B model at 16-bit precision that works out to about 128 KB per token, so a full 128,000 token context window needs roughly 16 GB: about as much as the model's own weights.

  5. Tricks to shrink it

    Multi-query and grouped-query attention let many attention heads share the same keys and values, cutting the cache several times over. Serving systems such as vLLM store the cache in small pages to avoid wasted memory. Prompt caching keeps a cache alive between requests so a long, repeated prompt is not read again.

Used during
inference only, when a model writes text. Training processes whole passages at once and does not need it
Sharing keys and values
Multi-query attention, Noam Shazeer at Google, 2019. Grouped-query attention, Google Research, 2023
Llama 3 example
8 key and value heads shared by 32 query heads, chosen "to reduce the size of key-value caches during decoding" (Meta, 2024)
Memory waste fixed
PagedAttention (vLLM, 2023) reported 2 to 4 times more throughput by managing cache memory in pages

Our 128 KB and 16 GB figures are our own arithmetic from Llama 3.1 8B's published settings: 32 layers, 8 key and value heads of 128 numbers each, 2 bytes per number.

Questions people ask

What does KV stand for in KV cache?

Key and value. In attention, every token produces a key (what it offers to be matched against) and a value (the information it passes on). The cache stores these two for every earlier token so they are not computed again.

Why does a long chat get slower or cost more?

Each new word still has to be compared with every stored note, and the notes take memory for every token in the chat. Longer chats mean a bigger cache, more memory per user and more work per word, so they cost more to run.

Is the KV cache the same as the chatbot remembering me?

No. The KV cache is temporary working memory for one conversation, kept only while it is being used (or for a few minutes with prompt caching). Remembering facts about you across chats is a separate feature, AI memory, which saves notes as text.

What is the difference between a KV cache and prompt caching?

The KV cache works inside one reply, word by word. Prompt caching keeps the KV cache of a long, repeated opening (like a big document or instructions) for a few minutes, so the next request can skip re-reading it.

Does the KV cache change the answer?

No. It is a shortcut that gives the same result as recomputing everything, just faster. Some memory-saving variants, like storing the cache at lower precision, can change results slightly.

Further reading

Prepared by Mundhra Labs. Reviewed by Aditya Mundhra. Checked October 2026.