What is a batch API (in plain English)?
Plain-English definition · Updated 2026-10-30. Discounts are stated as multiples of each provider's own list price, so they survive a price cut; the dollar examples are our arithmetic on published list rates, not vendor claims.
AITop is an independent guide. Prices and features change fast — always check the vendor's page.
A batch API is a queue you rent instead of a server you call. You hand over a file of independent requests, the provider runs them whenever it has spare capacity, and you get the answers back as a file — usually within an hour, contractually within 24 hours — for about half price. In 2026 that half price is industry standard: OpenAI quotes a 50% cost discount up to 50,000 requests / 200 MB per batch, Anthropic quotes 50% of standard prices up to 100,000 requests / 256 MB, and Google prices Gemini batch at 50% of the interactive rate with a 2 GB input file. The part nobody puts in the headline is the order of operations against prompt caching: OpenAI's price table halves cached input too (a 0.1× cached read becomes 0.05×), Google's documentation says the cache-hit discount takes precedence so the two do not compound, and Anthropic says they stack but that hits are best-effort, 30–98% because concurrent processing spreads identical prefixes across machines. Same 50% sticker, three different bills.
What it actually means
A normal API call is a promise about latency: you send a request and something is scheduled to answer it within seconds. A batch call is a promise about price instead. You upload a JSONL file where every line is a complete request with its own custom_id, the provider drops the whole file into a background queue, and when it is done you download a results file and join the rows back on that ID. Results come back in arbitrary order — every provider documents this — which is why the ID exists and why "match on position" is the first bug everyone writes.
The model is the same model. This is not a smaller, distilled or quantized variant: the batch endpoint runs the same weights with the same context window and, in Anthropic's case, the same tool use, vision and extended thinking. What you give up is latency and interactivity — no streaming, no partial results, no "how far along is it". OpenAI's completion_window currently only accepts 24h; Anthropic's ceiling is 24 hours with most batches finishing under one; Google's batch mode targets a 24-hour turnaround. You are not buying a worse model, you are buying a worse queue position.
The economics are the point. Inference providers run a demand curve with a hard peak, and batch work is the classic off-peak filler: it can be scheduled around interactive traffic, it does not need to hold a GPU warm for a user, and it fails cheaply. A 50% discount is roughly what that flexibility is worth to them. It is also why batch discounts tend to be flat across the whole model line rather than tiered — the saving comes from when the work runs, not from which model you picked.
Why it matters when you're picking a tool
The discount has an order of operations, and that order is the whole decision. Two levers now sit on the same invoice: batching (50% off) and prompt caching (a repeated prefix read at a tenth of input). If both compound, a cached token inside a batch costs a twentieth of list. If they do not, batching only touches the uncached remainder. Take a monthly workload of 500M input tokens at a $2-per-million base rate with a 90% cache-hit rate, and run the three published rules through it (our arithmetic, output tokens excluded — those simply halve):
- No batch, with cache: 0.19× list. On our example, about $190.
- Batch that compounds (OpenAI's table): 0.095× list — $95, half of the above, because the 0.1× cached read is itself halved to 0.05×.
- Batch that does not compound (Google's stated rule): 0.14× list — $140. The 90% cache discount wins and the batch discount only applies to the 10% of tokens that missed.
Same advertised 50%, a 32% difference in the bill. At a 0% hit rate all three arrive at the same 0.5×; the divergence grows with your hit rate and is at its widest — a factor of two — for a workload that is nearly all repeated prefix.
Anthropic answers the same question with a caveat instead of a rule. Its documentation says the batch discount and prompt caching can be combined, then warns that because batches are processed concurrently, cache hits are best-effort and typically land between 30% and 98%, with two pieces of advice that follow: mark the same cache_control block in every request, and use the 1-hour cache duration rather than the 5-minute default, since a batch routinely outlives five minutes. Translated into budgeting: do not model a batch pipeline at a 90% hit rate. Model it at a third and treat the rest as upside.
Limits decide whether your job fits at all. OpenAI takes 50,000 requests and 200 MB per batch, allows 2,000 batch creations per hour, and runs batch traffic on a separate rate-limit pool that does not consume your per-model token quota — which is the underrated feature here, because a backfill no longer competes with production. Anthropic takes 100,000 requests or 256 MB, keeps results downloadable for 29 days from creation, and does not bill requests that expire. Google takes a 2 GB file. If your job is 4 million documents, all three mean splitting — and splitting means the 24-hour window is per batch, not per job.
The 2026 reality check
Batch is no longer the only way to buy the 50%. OpenAI now sells flex processing — same token price as the Batch API, with prompt-caching discounts on top — in exchange for slower responses and occasional 429 Resource Unavailable errors that are not billed. That is a different product: flexible latency rather than a 24-hour file, which is what you actually want if a human is eventually waiting. Google's optimisation guide describes a comparable discardable-capacity tier at 50% of standard, explicitly aimed at multi-step agentic workflows where call N+1 depends on call N. The lesson is to stop treating "batch" as a synonym for "cheap" and start treating it as one of three price tiers — real-time, flexible, queued — that differ only in how much latency risk you absorb.
The real cost of batch is iteration, not tokens. There is no partial return, so a prompt bug costs a full cycle rather than ten seconds: you wait 24 hours to learn that your JSON schema had a typo. Every practitioner write-up of the last two years converges on the same workaround — validate on a small synchronous sample, freeze the prompt, then submit the bulk job. The discount only pays off on prompts you have stopped editing. Budget one synchronous week per pipeline before you move it to the queue, and keep the eval suite real-time: eval results you cannot iterate on are a museum piece.
Some vendors sell time-of-day instead of batching. DeepSeek has no batch endpoint; it sells the same flexibility through a peak/off-peak split where off-peak rates are half of peak, and peak is a narrow window — 01:00–04:00 and 06:00–10:00 UTC, Monday to Friday, excluding Chinese public holidays — leaving the rest of the week, including all weekend, at the lower rate. If your pipeline can simply shift rather than queue, that is a cleaner lever than a 24-hour file, because you keep streaming and tool use. The catch is identical in spirit: you are trading scheduling control for price, and the moment your workload becomes interactive the discount disappears.
Do not confuse a discount with a cheaper model. Halving a frontier model's rate still leaves it several times the price of a small model at full rate, and on most classification and extraction work the small model wins outright before either discount applies. The right sequence is: pick the cheapest model that passes your eval, then apply caching to its repeated prefix, then move the non-urgent half of the traffic to the queue. Teams that start from "how do I get 50% off" usually optimise the wrong term. The same trap shows up in inference cost generally: the headline rate is rarely the bill.
Quick checklist
- Ask which discount applies first. "50% off" plus "cached input at a tenth" is 0.05× on one provider and 0.1× on another. Model your own hit rate both ways before you commit; the gap at 90% hits is a third of the input bill.
- Freeze the prompt before you queue it. No partial results and no progress bar mean one bad schema costs a full 24-hour cycle. Run a synchronous sample first, then submit the bulk file.
- Size the job against the per-batch ceiling. 50,000 requests / 200 MB, 100,000 / 256 MB, or a 2 GB file — and a batch creation limit of 2,000/hour on OpenAI. Split deliberately, and remember the window is per batch.
- Keep eval suites and anything user-facing out of batch. The discount is for work where nobody is waiting; a chatbot, a coding assistant or a chained agent step all fail the "does anyone wait within the hour" test.
- Join on custom_id, never on position. Every provider returns results in arbitrary order. Persist the ID mapping before you submit, not after you download.
- Check whether flex beats batch for your case. If latency can stretch to minutes rather than a day, a flexible tier at the same token price keeps streaming and tool calls — and unbilled 429s beat a silent 24-hour wait.
Where you'll hit it
Three buying decisions on this site are really batch decisions: the best AI tools for small business (where "run this over last month's invoices" is a batch job wearing a SaaS subscription), ChatGPT vs Claude (both providers sell the same 50%, with different limits and a different caching rule underneath), and the best AI coding assistants — the explicit counter-example, since an agent loop that waits on its own last output can never be batched. Related terms: prompt caching is the discount batch has to be ordered against; AI token is the unit both meters count; KV cache is why a repeated prefix is cheap to re-read in the first place. Full list on the AI tools glossary.
The bottom line
A batch API is the least glamorous discount in AI and the easiest one to get wrong. The 50% is real and it is universal — OpenAI, Anthropic and Google all publish it, with limits of 50,000 requests/200 MB, 100,000/256 MB and 2 GB respectively — but the number you should budget is the combined rate against prompt caching, and that is where the providers diverge: compounding on OpenAI (cached input to 0.05×), non-compounding on Gemini (the cache discount takes precedence), best-effort on Anthropic (30–98% hit rates). On a 90%-cache-hit workload our arithmetic gives 0.095×, 0.14× and somewhere in between. So the four questions are: does the discount compound with caching, how big is one batch, what happens to requests that expire at the 24-hour mark, and is there a flexible tier at the same price that keeps my streaming. Answer those and batch stops being a coupon and becomes a scheduling decision — which is all it ever was.