Home › Glossary

What is tokens per second (in plain English)?

Plain-English definition · Updated 2026-10-27. The phase distinction below is standard across serving-stack documentation (vLLM's tuning guide treats prefill as compute-bound and decode as memory-bound) — but any specific tok/s number you read is hardware-and-model specific, so check what was measured.

AITop is an independent guide. Prices and features change fast — always check the vendor's page.

The 30-second answer

Tokens per second (tok/s) is the speed a language model writes text — and the label hides two different speeds. Prefill reads your entire prompt in one parallel pass and is compute-bound: it decides your time to first token (TTFT), which is what a long document costs you. Decode then writes the answer one token at a time and is memory-bandwidth-bound: every new token means streaming the model's weights out of memory again, so single-stream decode speed is roughly bandwidth ÷ model size in bytes. That formula explains most of what you feel: a 70B model at FP16 (~140 GB) tops out around 24 tok/s even on an H100 (3.35 TB/s), while a 7B model at 4-bit (~4 GB) can theoretically pass 250 tok/s on an RTX 4090. Batching makes a server serve more people — it does not make any one person's text stream faster.

What it actually means

A token is a chunk of text — roughly a word or part of one (see AI token, where the same unit is also the billing unit). Inference runs in two phases, and serving-stack documentation is explicit that they hit different hardware limits. Prefill processes all of your prompt tokens together — system prompt, chat history, retrieved documents, everything. Because the work is one wide parallel pass, the limit is raw math throughput, which is why serving docs describe prefill as compute-bound. Decode is a serial chain: each new token depends on all previous ones, and each step re-reads the model's weights from memory — which is why the same documentation describes decode as memory-bound.

The two phases report wildly different numbers for the same model on the same device — thousands of tok/s for prefill, tens to low hundreds for decode — and a benchmark line like "2,000 tok/s" that doesn't name its phase is usually quoting prefill, which is not the number you feel while reading. The three measurements that matter: TTFT (time to first token: queue wait + prefill — what a long prompt costs you), ITL / TPOT (inter-token latency, the gap between tokens once streaming starts), and throughput (tokens per second across all users on the machine — the number providers optimize, not the number you experience).

Why it matters when you're picking a tool

Because the decode formula is one line and it explains almost every speed difference between hardware: decode tok/s ≈ memory bandwidth ÷ bytes read per forward pass. For a dense model at batch size 1, the bytes per pass are roughly the size of the weights. NVIDIA's spec pages put the H100 SXM at 3.35 TB/s of memory bandwidth; an RTX 4090 is 1,008 GB/s and an RTX 4070 504 GB/s. Run the division (this is our arithmetic, not a vendor claim — real runs land below the ceiling because of KV-cache traffic and runtime overhead):

70B model at FP16 (~140 GB of weights) — H100 SXM: 3,350 ÷ 140 ≈ 24 tok/s ceiling; RTX 4090: ≈ 7 tok/s; RTX 4070: ≈ 3.6 tok/s. This is why big models need datacenter silicon to feel responsive before anyone mentions compute. 7B model at Q4 (~4 GB) — RTX 4090: ≈ 250 tok/s ceiling, comfortably fast. Same card, same day, 30–70× apart — the model choice, not the marketing, decides the speed.

Three levers fall out of the formula, and every one of them has its own entry on this site. Quantization (see quantization) shrinks the bytes per pass — a 4-bit file reads roughly a quarter of the data of FP16, which is why it speeds up decode specifically. Mixture of experts (see mixture of experts) means only the active parameters are read per token, so the bytes per pass are far smaller than the total parameter count suggests. And memory bandwidth is a spec-sheet number you can compare directly across cards — it predicts single-stream speed far better than TFLOPS does, because TFLOPS governs prefill, not decode.

The 2026 reality check

Batching makes servers richer, not your chat faster. Serving engines like vLLM pack many users' decode steps into shared batches, which raises total throughput enormously — but each individual stream still waits on the same memory-bound step. The throughput-vs-latency tradeoff is the central tension of LLM serving: you cannot maximize both, and a provider quoting "up to X tok/s" is almost always quoting aggregate throughput under load, not what your session will feel like. Chunked-prefill scheduling (which batches compute-bound prefill work with memory-bound decode work) exists precisely to protect ITL from large prompts — the serving docs treat that interference as the default failure mode, not an edge case.

Every speed claim needs a scope or it cannot be checked. Which phase (prefill or decode)? Which model, at which quantization? Which runtime (llama.cpp, vLLM, TensorRT-LLM), which device, batch size 1 or under load? Community benchmark tooling separates the phases for exactly this reason — llama.cpp's llama-bench reports prompt processing (pp) and text generation (tg) as different numbers. A claim scoped this narrowly survives replication; a bare "2× faster" does not. The same skepticism applies to "instant" AI tool demos: a snappy reply to a short question says nothing about how the tool handles your 30-page document, because that wait is prefill (TTFT), which the demo never showed.

Context length taxes decode too, not just TTFT. Every decode step also reads the growing KV cache — the attention state for your whole conversation — so long-context sessions get slower per token even when the weights fit comfortably (see context window). If a tool feels fast on message one and sluggish by message twenty, that is the cache growing, not your imagination.

Quick checklist

Where you'll hit it

The two-speeds split explains three purchases on this site: the Ollama vs LM Studio comparison (both are local runners, and the tok/s you get is decided by the formula above, not the UI), the best AI chatbots ranking (where a tool that feels instant on short chats can stall on long RAG documents — that is prefill, and it bills as inference cost too), and quantization (the lever that shrinks the bytes per pass). Related terms: mixture of experts cuts the bytes read per token architecturally, and context window is the budget whose KV cache taxes every decode step. Full list on the AI tools glossary.

The bottom line

Tokens per second is two numbers wearing one label: prefill, which is compute-bound and sets your time to first token, and decode, which is memory-bandwidth-bound and sets how fast the text streams. The only formula a buyer needs is decode speed ≈ memory bandwidth ÷ model size in bytes — it puts a 70B FP16 model near 24 tok/s even on an H100 and a 7B 4-bit model past 250 tok/s on an RTX 4090, and it tells you that bandwidth, quantization, and active-parameter count matter more than TFLOPS for anything you actually read. Be suspicious of any tok/s claim that doesn't name its phase, model, quant, runtime, and hardware; be extra suspicious of "up to X tok/s" from a provider, because that is batched throughput for their server, not a speed you will ever feel. Two numbers, one question each: how long until the first token (TTFT), and how fast does it type (ITL).