What is speculative decoding (in plain English)?
Plain-English definition · Updated 2026-10-31. Speedup figures are quoted from the primary papers and official project pages, with the baseline hardware named; the per-cycle arithmetic is ours.
AITop is an independent guide. Prices and features change fast — always check the vendor's page.
Speculative decoding is a way to make a large language model generate text faster without changing what it generates. A small, fast draft model guesses the next several tokens; the big target model checks all of them in a single forward pass — roughly the cost of generating one — keeps the ones that match, and resamples the first one that does not. The acceptance rule is a modified rejection sampling that provably preserves the target model's output distribution, so the text is statistically identical to ordinary decoding: this is why the technique is called lossless. The two papers that introduced it in 2022–23 measured 2–3× on T5-XXL "with identical outputs" (Leviathan et al., Google) and 2–2.5× on Chinchilla 70B (Chen et al., DeepMind); the EAGLE-3 project's official page now claims 5.6× over vanilla decoding on Vicuna 13B. The catch: it is a serving-side technique. It buys latency for whoever runs the model — and it never appears as an option or a discount on a hosted API bill.
What it actually means
Normal decoding is strictly serial: token 2 cannot be computed until token 1 exists, because each token's input includes the previous one. As our tokens-per-second entry explains, that makes decode memory-bandwidth-bound — every step reads every model weight off VRAM once, whether it produces one token or one thousand. Speculative decoding exploits the gap between the two: scoring several candidate tokens in parallel costs about the same as scoring one, because the weights are read once either way. So you spend a cheap model's time guessing, and the expensive model's single pass confirming.
The part that makes it rigorous rather than a heuristic is the acceptance rule. If the draft's token is more probable under the target than under the draft, it is always accepted; if it is less probable, it is accepted with a probability proportional to the gap, and a rejected position is resampled from the corrected residual distribution. Leviathan, Kalman and Matias (Google Research, arXiv 2211.17192) and Chen et al. (DeepMind, arXiv 2302.01318) published this scheme independently within weeks of each other in 2022–23, and both papers prove the same theorem: the resulting sequence is drawn from exactly the target model's distribution. Chen et al.'s phrasing is the careful one — "preserves the distribution of the target model within hardware numerics" — the floating-point caveat every honest benchmark carries.
How much you gain per pass has a formula, and it only needs one number: the acceptance rate α — how often the draft agrees with the target. The expected tokens emitted per target pass is (1 − αγ+1) / (1 − α), where γ is the number of drafted tokens (Leviathan et al.'s expression). Our arithmetic at α = 0.8 and γ = 5: (1 − 0.86) / 0.2 ≈ 3.7 tokens per pass, which at a 4–5× cost ratio between the models is where the headline 2–3× speedups come from. The formula also shows the failure mode: at α = 0.5 the same γ = 5 draft yields 1.97 tokens per pass, barely above break-even once the draft's own latency is counted. The draft's agreement rate is the whole product.
Why it matters when you're picking a tool
If you self-host, this is the highest-leverage knob after quantization. Quantization (see our entry) changes the weights to make each pass cheaper; speculative decoding keeps the weights and removes passes. The method families, in rough order of effort: an independent draft model (a small model from the same tokenizer family — extra VRAM, moderate acceptance); n-gram / prompt lookup (no model at all, matches repeated spans from the prompt — near-zero cost, works best on code editing, summarization and other echo-heavy output); Medusa-style heads (extra decoding heads trained onto the target model itself); and EAGLE-style feature drafters, which draft from the target's own hidden states. The EAGLE project's official page reports the progression on Vicuna 13B: 3× (EAGLE), 4× (EAGLE-2), 5.6× (EAGLE-3) versus vanilla decoding, on 2× RTX 3090s — and cites a third-party evaluation naming it the fastest speculative method.
Check whether your target model ships its own drafter. Multi-token-prediction (MTP) heads trained into the model itself have become a native draft source — DeepSeek's V3 lineage popularized the approach, and vLLM exposes it as the built-in mtp method alongside eagle, eagle3, ngram, draft_model and others. If native MTP weights exist for your model, that is usually the lowest-friction path: no second model to serve, no matching checkpoint to hunt for.
Now the caveat that decides everything: latency, not throughput. vLLM's own documentation scopes speculative decoding to "low to medium QPS, memory-bound workloads" — one user waiting on one answer, where inter-token latency is the product. Their method-selection table rates EAGLE-class methods high-benefit at low QPS but only medium-to-high at high QPS, and notes that the draft-free methods (n-gram, suffix) earn "modest speedups without increasing workload during peak traffic". Under a saturated batch, spare bandwidth is already consumed by other requests, the draft becomes pure overhead, and the win shrinks toward zero. If you are buying tokens by the million rather than serving one user, this is not your lever.
The 2026 reality check
You cannot order it from an API. Speculative decoding is something the operator of a model does; as an API buyer you see none of it — no parameter to set, no discount to claim. That is also why a hosted frontier model's tokens-per-second has quietly improved without any model change announcements: serving-side speedups like this land in the provider's cost structure, and show up in your bill only indirectly, if at all. When you compare AI coding assistants on response feel, part of what you are feeling is this class of engineering, not the model.
Lossless does not mean better. The output distribution is provably identical, which means speculative decoding makes the model you already chose faster and nothing else. It will not fix a weak model, reduce hallucinations, or change reasoning quality — if the model was wrong at 20 tok/s it is wrong faster at 50. Evaluate the model first, then apply this. The one quality-adjacent trap is practical rather than statistical: a misconfigured draft setup (wrong tokenizer family, unvetted community draft checkpoints) can silently degrade acceptance rates back to break-even, so measure the acceptance metric each engine exposes rather than trusting the config.
The 5–6× numbers come with a named baseline, so read them like one. EAGLE-3's official 5.6× is measured on Vicuna 13B at fp16 on two RTX 3090s — a specific, modestly sized model on consumer hardware — and the project itself tells benchmarkers to compare against official checkpoints and official draft-tree setups. Smaller targets leave thin margins (a 7B model's draft overhead eats the win), and every published speedup is batch-size dependent. Treat any speedup figure without its hardware and batch context the way you would treat a tok/s benchmark without a model name: as marketing.
It composes with everything else on this site. Speculative decoding stacks with quantization (quantize the draft even harder than the target), with KV-cache management (each rejected draft token still writes KV entries that must be unwound), and it is orthogonal to prompt caching, which attacks the prefill side that speculative decoding never touches. If your workload is long documents and short answers, caching is the bigger lever; if it is a single user watching text stream, this one is.
Quick checklist
- Locate yourself on the QPS curve first. One user waiting on one answer is the target regime; a saturated batch server is where the technique pays nothing. vLLM's documentation says this in so many words — check the traffic pattern before the method.
- Measure acceptance rate, not vibes. Expected tokens per pass is (1 − αγ+1)/(1 − α) in the draft's agreement rate α; at α = 0.8 you clear ~3.7 tokens per pass, at 0.5 you are near break-even. Every major engine exposes per-request acceptance metrics — log them from day one.
- Try the zero-weight methods before training anything. N-gram / prompt-lookup drafting needs no model, no VRAM and no training, and is disproportionately effective on output that echoes the input: code edits, summarization, structured rewrites. Minutes to test.
- Check for native MTP weights on your target model. If the model family ships its own multi-token-prediction heads, that is the lowest-friction high-performance path — no second model, no tokenizer compatibility risk.
- Read every speedup number with its baseline. Model size, hardware, precision, batch size. "5.6× faster" and "2–3× on T5-XXL with identical outputs" are claims about different worlds; neither transfers to yours without measurement.
- Do not expect quality changes. The distribution is provably identical — this is a speed product. If the model needs to be better, change the model; if it needs to be cheaper per token, see inference cost.
Where you'll hit it
Two decisions on this site are really speculative-decoding decisions. Ollama vs LM Studio is the local-serving matchup where a drafter is easiest to try — point your engine at a matching draft checkpoint and measure inter-token latency on your own hardware. And the best AI coding assistants are the felt-experience case: coding output echoes the prompt heavily, which is exactly the pattern n-gram drafting eats, so part of an assistant's snappiness is this technique rather than the model behind it. Related terms: tokens per second explains why decode is the serial bottleneck this technique breaks; KV cache is the memory bill every rejected draft token briefly inflates; quantization is the other half of the local-speed playbook. Full list on the AI tools glossary.
The bottom line
Speculative decoding is the rare acceleration that comes with a theorem: draft several tokens, verify them in one pass, accept by the rejection-sampling rule, and the output distribution is the target model's — full stop. The measured returns are real (2–3× in the founding papers, 5.6× claimed by EAGLE-3 with its baseline named) and the technique is now a standard switch in every major serving engine. But it is a serving-side lever with a specific regime: one waiter, spare bandwidth, an echo-heavy workload. It never shows up on a hosted API bill, it will not make a weak model better, and under a saturated batch it pays nothing. If you self-host, measure the acceptance rate and start with the free drafters. If you buy tokens by the million, file this one under "why the operator's margin improved" — and go negotiate your own discount.