Home › Glossary

What is inference cost?

Plain-English definition · Updated 2026-10-21. Numbers dated; verify with the vendor.

AITop is an independent guide. Prices and features change fast — always check the vendor's page.

The 30-second answer

Inference cost is what you pay to run a model — every question answered — as opposed to the one-off cost of training it. It is not one number, because the meter is lopsided in four places. Output bills at five to six times the input rate on both major API vendors. Cached input bills at five to ten percent of the fresh rate. Batch jobs cost half and priority jobs cost double. And context length is a cliff, not a slope: cross a vendor's threshold and the same token gets more expensive mid-conversation. Self-hosted, the shape is different but the lesson is the same — cost per token is the GPU's hourly price divided by the throughput it actually sustains, so at batch size one a card can cost forty times more per token than it does under load. In both worlds, the bill is set by your traffic shape, not by the headline price per million.

What it actually means

Every model has two lives. Training is the expensive, one-time run that produces the weights. Inference is every use after that: the forward pass that turns your prompt into an answer. Inference cost is the second one, and it is the one that compounds — a training run happens once and then sits on the balance sheet, while inference happens millions of times and shows up on every invoice until you switch the product off.

The hosted version of the bill is simple arithmetic: input tokens × input rate + output tokens × output rate, per million. What makes it hard to predict is that the two rates are not the same, and neither is constant. The self-hosted version replaces that meter with a division: cost per token = cost of the hardware per hour ÷ tokens produced per hour. Same idea, different lever — and the lever is utilisation, because a GPU that is idle still costs the same per hour as one that is saturated.

That is why two teams running "the same model" can pay wildly different amounts. They are not buying the same thing. One is buying tokens at a price list; the other is buying GPU-hours and hoping to fill them. Inference cost is the term that forces those two facts into one conversation, which is exactly what a procurement decision needs.

Why it matters when you're picking a tool

Because the ratio between input and output decides which architecture is cheaper, and most buyers never check it. Output tokens are where the money is: on OpenAI's current price sheet a short-context gpt-6.1-sol request bills $2.00 per million input and $10.00 per million output, and the same five-to-one shape holds across its siblings — gpt-5.4 at $2.50 and $15.00, gpt-6-astra at $10.00 and $50.00. Anthropic's current line is identical in structure: Sonnet 5.5 at $2 / $10, Opus 5.5 at $4 / $20, Haiku 5.5 at $0.10 / $0.50. Five times, every model, both vendors. If your product generates long answers from short prompts you are living on the expensive side of the meter; if it classifies, extracts or reranks — long input, one-line output — you are on the cheap side. Same vendor, same model, opposite economics.

Three multipliers then decide the rest, and they all reward a boring property: repetition. Cached input is billed at a fraction of fresh input — $0.10 against $2.00 on gpt-6.1-sol, $0.10 against $2.00 on Sonnet 5.5, $0.01 against $0.10 on Haiku 5.5 — so a chatbot that resends the same system prompt and history every turn is paying full price for text the model has already seen. Batch processing is half price on both vendors for work nobody is waiting on. Priority processing is double price on both for work somebody is. None of these are discounts you negotiate; they are modes you choose, and choosing none of them is itself a decision to pay list price.

So the buyer's question is not "which model is cheapest per million tokens". It is "what is the shape of my traffic, and which of these four levers does it touch". A support bot with a 4,000-token system prompt, a 1,200-token conversation and a 200-token answer is a caching problem wearing a model-choice costume. A nightly document pipeline is a batch problem. A live agent that must respond inside 800 ms is a priority problem, and it will cost double no matter which vendor wins the bake-off.

The 2026 reality check

The published rates moved, the shape did not. The absolute numbers on both price sheets are lower than a year ago and will be lower again; the five-to-one output premium has survived every generation so far, because it reflects physics rather than pricing strategy. Generating a token requires reading the whole model and writing one token's worth of work; checking a prompt token can be done in parallel with thousands of others. Output is the serial part, and serial work does not get cheaper just because the parallel part did. A vendor that quoted input-only prices and hid the output rate would be quoting half a product.

Context length is a cliff. This is the 2026 detail that most cost models still miss. OpenAI's current sheets split at roughly 272K input tokens, where input doubles and output rises by half — gpt-6-astra goes from $10 / $50 to $20 / $75 per million. Anthropic's Haiku 5.5 splits at 100K tokens, and above that line input goes from $0.10 to $0.50 and output from $0.50 to $2.50 — five times, for the same model. A retrieval pipeline that grows its context from 90K to 110K tokens does not get 20% more expensive; on that model it gets five times more expensive. Design to sit under the line, or budget for the jump deliberately.

Caching is the biggest lever nobody prices. It is also the most fragile: OpenAI's cache starts at a 1,024-token prefix, extends in 128-token increments, and is typically cleared after five to ten minutes of inactivity and always within an hour of last use. Anthropic prices a five-minute cache write at 1.25× input, so a cache you write and never read is a loss. Warm caches reward steady traffic and punish bursty traffic — the same product can see a 40% saving in one week and none in the next without a line of code changing.

Self-hosting trades a price list for a utilisation problem. Token generation is memory-bandwidth-bound, not compute-bound: each new token requires streaming the weights past the compute units, which works out to roughly one FLOP per byte against a hardware balance point of about 206 FLOP/byte on an H200. The practical ceiling is bandwidth divided by the bytes read per token — a 70B model in FP8 is about 70 GB, so 4.8 TB/s ÷ 70 GB ≈ 69 tokens per second on one H200 at batch size one, against about 48 on an H100's 3.35 TB/s. Now divide the rental rate by that throughput: a card at $2.50 an hour producing 48 tokens a second is about $14.50 per million output tokens; the same card batching enough concurrent users to sustain 2,000 tokens a second is about $0.35. Forty times the difference, from batching alone, with no change in hardware or model.

And the arithmetic is the only honest forecast. Vendors are entitled to reprice; both sheets above carry dated promotions and tier changes. Treat every rate in this entry as an observation from October 2026, recompute with your own traffic mix, and re-check before you sign anything multi-year.

Quick checklist

Where you'll hit it

Inference cost is the number behind three arguments you will meet in any AI tool purchase: the best AI chatbots of 2026 (where "unlimited" plans are priced against a token cost the vendor is quietly absorbing), the best AI coding assistants (where long files and repeated context make caching the difference between a viable seat price and a loss-making one), and the best AI customer service tools (where per-resolution pricing is an inference bill with the variance moved to the vendor). Related terms: AI token is the unit this cost is metered in; context window explains why the cliff exists at all; AI agent is the workload that multiplies every lever at once, because an agent makes many calls per task; and fine-tuning is the decision that adds a permanent premium to this bill. Full list on the AI tools glossary.

The bottom line

Inference cost is the price of every answer, and it is set by four levers the price list does not advertise: output costs five to six times input, cached input costs a tenth, batch halves it, priority doubles it — with context length acting as a cliff rather than a slope. Self-hosted, the same logic arrives as a division, and utilisation becomes the multiplier that can move cost per token by 40×. So the buying question is never "what does a million tokens cost". It is "what shape is my traffic, which levers does it touch, and where is the nearest cliff" — and the team that answers that pays a fraction of what the team beside it pays for exactly the same model.