Home › Glossary

What is a KV cache (in plain English)?

Plain-English definition · Updated 2026-10-28. The size formula below is standard across inference-serving literature — vLLM's own engineering blog treats KV-cache management as the central memory problem of LLM serving — but any specific cache number is model-and-precision specific, so run the arithmetic for your model.

AITop is an independent guide. Prices and features change fast — always check the vendor's page.

The 30-second answer

The KV cache is where an inference server keeps the attention keys and values of every token you've already sent, so each new token doesn't recompute attention over the whole conversation. It is the quiet second consumer of your VRAM — and it grows linearly with context length, not with model cleverness. The buyer's formula: 2 × layers × KV heads × head size × bytes per element, per token. For Llama 3 70B (80 layers, 8 KV heads, head size 128, FP16) that is about 320 KiB per token — an 8K-token chat carries ~2.5 GiB of cache, and a 128K-context session carries ~40 GB, on par with the model's own weights. The cache is why long chats slow down, why "my model fits but it OOMs anyway" happens, and why providers charge differently for fresh vs cached input tokens.

What it actually means

A transformer generates text one token at a time, and every new token attends to all the tokens before it — your system prompt, the document you pasted, the whole conversation. During the forward pass, each layer computes a key and a value vector for every token. Since past tokens never change, recomputing their keys and values at every step would be pure waste, so the runtime stores them in GPU memory. That stored state is the KV cache. It is a deliberate trade: spend memory to skip redundant compute — the same memory-vs-recompute logic that shows up in quantization, just in the opposite direction.

The size arithmetic is one line, and it is worth memorizing because no vendor puts the answer on a spec sheet: cache bytes = 2 × layers × KV heads × head dimension × context length × batch size × bytes per element. The leading 2 counts the separate K and V tensors; everything else is the model's architecture. The number of query heads is deliberately absent — under grouped-query attention, many query heads share one KV head, which is the main architectural lever for shrinking this number (see mixture of experts for the other spec-sheet trap of two param counts).

Why it matters when you're picking a tool

Weights are the bill you planned for; the cache is the bill you get. Worked example, using numbers that are public and cross-checkable — Llama 3 70B: 80 layers, 8 KV heads, head size 128, FP16 cache. Per token: 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes ≈ 320 KiB. An 8,192-token chat: ~2.5 GiB. A 128,000-token document session: ~40 GB — roughly the size of the model's FP8 weights. That is why the most common local-LLM failure is not "the weights don't fit" but "the weights fit at 2K context and the process dies at 32K" (see context window). If a local runner like the ones in our Ollama vs LM Studio comparison crashes on your long document, the KV cache is the usual suspect.

The cache, not the weights, sets how many people a GPU can serve. A server keeps every active conversation's cache in memory, so the batch size ceiling is (memory left after weights) ÷ (cache bytes per conversation). This is why architecture choices that shrink the per-token cache translate directly into capacity: Llama 3 70B uses 8 KV heads instead of 64 — an 8× reduction from grouped-query attention alone; KV-cache quantization to FP8 halves it again; and DeepSeek's multi-head latent attention (MLA) compresses the cache into a smaller latent vector per token.

Serving engines exist largely to manage this one data structure. vLLM's original engineering writeup is blunt: its authors found existing serving systems wasting 60–80% of KV-cache memory to fragmentation and over-reservation, and built PagedAttention — which pages the cache like an operating system pages virtual memory — to cut waste to under 4%. The same block-table trick lets concurrent requests share the cache of a common prefix (system prompt, pasted document), which vLLM reports cuts memory use for shared-prefix workloads by up to 55%. Prefix reuse is also why API providers can bill cached input tokens at a fraction of fresh ones — the discount reflects real memory savings (see inference cost).

The 2026 reality check

Long context taxes speed, not just memory. Every decode step reads the entire accumulated cache, so per-token latency creeps up as the conversation grows even when everything fits comfortably — the effect we flagged in tokens per second. A tool that feels instant on message one and sluggish by message twenty is not degrading; its cache is growing. Advertised context windows describe what the model can hold, not what your hardware holds at full speed.

"1M context!" claims and usable context are different numbers. A model card quoting a maximum context length says nothing about how much VRAM the cache consumes there, how much the first token will cost, or how degraded attention is at the far end. In 2026 the honest question for any long-context pitch is: at that length, what is the cache footprint, and is it quantized? A 128K session on a 70B model at FP16 cache is a ~40 GB structure before you serve a second user. Marketing quotes the window; the arithmetic quotes the bill.

The cache is also the reason providers price context at all. Per-token pricing exists because serving memory scales with stored context, and the cached-vs-fresh input price split exists because prefix caching turns that memory into a shared, reusable asset. When you evaluate an API or a self-hosted stack on a ranking like our chatbot guide, remember that the same model can be cheap with prompt reuse and expensive without it — the KV cache is the mechanism behind both numbers.

Quick checklist

Where you'll hit it

The cache explains three purchases on this site: the Ollama vs LM Studio comparison (both runners expose context-length and KV-quantization settings, and those settings decide whether your long document OOMs), the best AI chatbots ranking (where "handles long documents" claims are really claims about cache management and pricing), and quantization (which shrinks both weights and cache, but by different dials). Related terms: context window is the budget the cache spends against; tokens per second is the speed the cache taxes; mixture of experts shrinks the bytes read per token while the cache shrinks the bytes stored. Full list on the AI tools glossary.

The bottom line

The KV cache is the attention state of your entire conversation, stored so the model never recomputes it — and it is the second consumer of your VRAM that no spec sheet mentions. One formula covers it: 2 × layers × KV heads × head size × bytes, per token, per conversation. That arithmetic puts a Llama-3-70B-class chat at ~320 KiB per token — 2.5 GiB at 8K context, ~40 GB at 128K — and it explains the three things buyers actually feel: why long chats OOM even when the weights fit, why every provider caps concurrency, and why cached input tokens bill at a fraction of fresh ones. Check KV-head count on the model card, budget cache alongside weights, and treat any context-window headline as a memory bill you haven't been shown yet.