Home › Glossary

What is reranking (and why does it bill you by the pair)?

Plain-English definition · Updated 2026-11-07. Model specs come from Cohere's Rerank documentation, the official Azure AI Foundry and Amazon Bedrock price pages, and Voyage AI's pricing documentation. The 65-hours-versus-5-seconds figure comes from the Sentence-BERT paper (arXiv:1908.10084) and the SentenceTransformers documentation. The per-query cost arithmetic is ours, and labelled as such.

AITop is an independent guide. Prices and features change fast — always check the vendor's page.

The 30-second answer

Reranking is the second stage of a two-stage search pipeline: a cross-encoder model reads your query and each candidate document together and returns a relevance score, re-sorting the candidates that a cheap first stage (usually an embedding search) already picked. It is more accurate than the vector search it sits on top of precisely because the two texts go through the model jointly — and that joint pass is why you cannot precompute it, and why the bill scales with candidates × queries, not with corpus size. The published numbers make the trade concrete: the Sentence-BERT paper measured that scoring every pair among 10,000 sentences with a cross-encoder takes about 65 hours on a V100, versus about 5 seconds with precomputed embeddings. As a service, Cohere sells it per search unit — the official Azure price page lists Rerank 4 Fast at $2.00 per 1,000 searches (up to 100 document chunks per unit) — while Voyage bills per processed token ($0.05 per million for rerank-2.5, with "processed" defined so that 100 candidates of 500 tokens each bill ~50,000 tokens per query). Run a million queries a month over 100 candidates and reranking costs $2,000–2,500 a month on either vendor — against a one-time embedding bill of about $10 for the same corpus.

What is reranking, exactly?

A reranker is a model that takes one query and a list of candidate documents and returns the same list sorted by relevance, each document carrying a score. It never sees your whole corpus. It sees the 100 (or 1,000) candidates your first-stage search already selected, and its whole job is to fix their order. The input is a query plus a candidate; the output is a number. There is no generated text, no prompt, and no vector coming back.

The architecture behind it is the cross-encoder, and the contrast with the bi-encoder (the embedding model) explains everything about cost and quality. A bi-encoder processes your query and each document separately, producing a vector for each, so document vectors can be computed offline once and compared cheaply at query time with a dot product. A cross-encoder processes the query and a document together in a single pass and outputs one relevance score. The SentenceTransformers documentation states it flatly: cross-encoders "achieve better performances than bi-encoders," but "do not produce a sentence embedding" and cannot be indexed — a query against a corpus becomes one full model pass per document.

That is why reranking exists as a stage rather than the whole search engine. The standard pipeline the same documentation recommends: retrieve the top 100 with an efficient bi-encoder, then rerank those 100 by scoring every (query, hit) combination. You pay the expensive joint read only on a shortlist. Every serious retrieval stack works this way, whether or not the product page names the reranker — see RAG for what happens with the reranked shortlist afterwards.

Why can't you just use the accurate model for everything?

Because the accurate model is O(candidates) full model passes per query, and that arithmetic does not negotiate. The Sentence-BERT paper by Nils Reimers and Iryna Gurevych (Technische Universität Darmstadt, EMNLP 2019, arXiv:1908.10084) put a number on it that has aged well: finding the most similar pair in a collection of 10,000 sentences requires about 50 million inference computations — roughly 65 hours on a V100 GPU — with the cross-encoder approach. Precomputing 10,000 sentence embeddings with SBERT and comparing them takes about 5 seconds, plus roughly 0.01 seconds of cosine similarity. That four-orders-of-magnitude gap is the entire business case for embeddings, and the reason the cross-encoder gets relegated to a shortlist.

The corollary matters for your bill: the reranker's cost is the compute you would have needed to search the whole corpus without one. When you buy a hosted rerank API, you are buying back a slice of those 65 hours at metered rates — per pair. Which is exactly how the two main vendors bill it.

What do the two main vendors actually sell?

Cohere is the incumbent and sells reranking per search unit: one unit is a query with up to 100 document chunks; a request with 350 documents is billed as 4 queries, and any document longer than 512 tokens gets split into chunks that each count as documents. The official Azure AI Foundry price page lists Cohere Rerank v4 Fast at $2.00 per 1K searches and Rerank v4 Pro at $2.50 per 1K searches; Amazon Bedrock lists the older Rerank 3.5 at $2.00 per 1,000 queries under the same 100-chunk rule. Cohere's own documentation shows the model line has since moved to Rerank 5 — rerank-v5.0-pro (about 2B parameters) and rerank-v5.0-fast (about 500M) over a 32,768-token combined query+document context, 100+ languages, and calibrated 0–1 relevance scores — with Rerank 3.5 remaining at a 4k context. Cohere's own pricing page no longer lists per-unit API rates (access routes through enterprise and Model Vault plans), so the durable public prices are the cloud-marketplace ones.

Voyage AI sells reranking per processed token, and its definition of "processed" is the load-bearing detail: (number of query tokens × number of documents) + sum of the tokens in all documents. rerank-2.5 costs $0.05 per million tokens and rerank-2.5-lite $0.02, with the first 200 million tokens free per account. Voyage's own worked estimate: a request with 100 documents where query plus each document totals 500 tokens bills about $0.0025 per request. A Batch API exists (12-hour window, 33% discount, free credits not applicable).

Vendor / modelBilling modelPublished priceContextThe catch
Cohere Rerank v4 FastPer search unit — 1 query ≤ 100 doc chunks$2.00 / 1K searches (Azure price page)—Docs over ~512 tokens split into chunks, each counting toward the 100 — your chunk count is your bill
Cohere Rerank v4 ProPer search unit$2.50 / 1K searches (Azure price page)—+25% over Fast for the quality tier
Cohere Rerank 3.5Per query, ≤ 100 chunks$2.00 / 1K queries (Amazon Bedrock)4k combinedOlder generation; still the Bedrock list price
Cohere Rerank 5 (pro / fast)See current marketplace listingsNot publicly listed on cohere.com32,768 combined~2B / ~500M params; adds Context Focus (returns relevant passages, not just order)
Voyage rerank-2.5Per processed token: (query × docs) + all doc tokens$0.05 / 1M tokens (≈$0.0025 per 100-doc request)—The formula multiplies candidates by query length — a 2× candidate budget is a 2× bill
Voyage rerank-2.5-litePer processed token$0.02 / 1M tokens—Lighter model, same meter; the obvious A/B default

Price note (2026-11-07): the Cohere v4 figures are from Microsoft's official Azure AI Foundry price page (azure.microsoft.com, "Cohere Rerank v4 Fast Global $2 / 1K searches, Rerank v4 Pro Global $2.50 / 1K searches"); the Rerank 3.5 figure is from Amazon's official Bedrock pricing page, which also documents the 100-chunk rule. Cohere's own pricing page routes model access through enterprise and Model Vault plans and does not publish per-unit API rates, so we cite the marketplace pages rather than inferring. Voyage figures are from Voyage's own pricing documentation; third-party trackers (LiteLLM, UsagePricing) match both vendors' numbers. Two vendors, two billing models, one sanity check: at 100 candidates of ~500 tokens, Voyage's token meter works out to ~$0.0025 per query — within 25% of Cohere's $0.002. The two meters agree, which is the best evidence either is honestly priced.

Why it matters when you're picking a tool

Because the embedding bill is one-time and the rerank bill is forever. Take a corpus of one million chunks at 500 tokens each. Embedding it once costs about $10 at OpenAI's text-embedding-3-small rate (see our embedding entry). Now rerank it: one million queries a month over 100 candidates is 1,000 search units × 1,000 = one million units at Cohere's rate — $2,000 a month — or roughly 50 billion processed tokens at Voyage's — $2,500 a month. Two independent billing models land within 25% of each other, so treat ~$2–2.5 per thousand queries (at 100 candidates) as the market price of accuracy, not a vendor quirk. The reranker is not a rounding error on the embedding bill; it is 200× the embedding bill.

Because "2¢ per query" is a human-scale number wearing an agent-scale problem. Ten queries a day is under a dollar a month — free, effectively. Ten thousand queries a day (300,000 a month — a busy support bot, an agent loop that searches while it works) is $600–750 a month per application, at 100 candidates, before you add a second rerank pass for a second data source. The 2026 pattern that breaks budgets is agentic retrieval: an agent that issues dozens of searches per task multiplies your per-query line by its iteration count, and the rerank meter multiplies with it.

Because your chunking strategy is a billing decision. Cohere counts chunks: over 100 chunks per request means multiple queries, and documents over 512 tokens get split into chunks that each count. The same corpus chunked twice as fine is roughly twice the rerank bill for the same retrieval job. Voyage's formula has the same shape — processed tokens scale with the number of documents — so doubling your candidate depth (reranking top-200 instead of top-100) doubles the bill while adding diminishing accuracy. Budget reranking as queries × candidates, and you will never be surprised by an invoice.

Because the accuracy is real and worth paying for — on the shortlist only. The research consensus (and the Sentence-BERT paper's own framing) is that cross-encoders outperform bi-encoders on relevance scoring; that is why the retrieve-then-rerank pattern exists. The rational deployment is a cheap, slightly oversized first stage (retrieve 100 with a small embedding model) and the best reranker you can afford on the top of the stack, with top_n set to the handful of chunks your context window can actually hold. Paying cross-encoder prices to score the whole corpus is the arithmetic error the SBERT paper was written to prevent.

The 2026 reality check

Rerankers are moving up the stack. Cohere's Rerank 5 adds Context Focus: the same rerank call can return the most relevant passages inside each document (character offsets with scores, off by default, enabled via a relevant_spans parameter), so the reranker stops being just an orderer and starts deciding what text reaches the LLM. Voyage's rerank-2.5 is instruction-following — a query can carry a prefix stating what "relevant" means for this search. The direction is consistent across vendors: the reranker is becoming the relevance policy layer of retrieval, not the last formatting step.

The pricing pages have gone dark in a specific way. Cohere's own site now routes model access to enterprise and Model Vault plans, leaving the cloud marketplaces (Azure, AWS Bedrock, SageMaker, OpenRouter) as the only public per-unit rates — and the marketplace generations lag the docs (Azure lists v4 while the docs headline v5). Practical rule: treat marketplace prices as a ceiling for list-price planning and expect enterprise discounts under them; and re-check the generation alignment before comparing "Cohere" as if it were one price.

What a reranker cannot fix has not changed. It reorders the candidates it is given. If the first-stage retrieval missed the right document entirely, the reranker scores garbage against garbage and returns it confidently — the same failure mode that makes RAG fail when the answer is not in the corpus. Relevance scores are not truth scores; a well-scored wrong document is still wrong, which is why hallucination needs its own defences regardless of how good your reranker is.

Quick checklist

Where you'll hit it

Every retrieval-backed product on this site either uses a reranker or is leaving accuracy on the table; you will meet it in three places. Support and knowledge tools — a customer service assistant that cites the right article in the top three but the wrong one first is the classic reranker-shaped failure, and the reason enterprise search vendors bundle one invisibly. Code search and repo agents — coding assistants and tools like those compared in Claude Code versus Cursor rerank retrieved functions against your task, and a repo-sized corpus makes the first-stage/reranker split mandatory rather than optional. And local stacks — if you run retrieval on your own hardware, a cross-encoder model competes with your LLM for the same VRAM budget that quantisation arithmetic governs, which is the hidden cost in setups compared in Ollama versus LM Studio. Related entries: embedding for the first stage this sits on top of, RAG for the pipeline around it, and token pricing for the LLM-side tokens a good reranker saves. Full list on the AI tools glossary.

The bottom line

Reranking is the rare AI line item where the physics, the architecture and the invoice all say the same thing: accuracy costs compute per pair. A cross-encoder reads your query and each candidate together, which is why it beats the embedding search underneath it and why it can never be free — Sentence-BERT's 65 hours versus 5 seconds is the whole story in two numbers. Buy it anyway, on three conditions. Keep the candidate count at 100; keep your chunk count honest (it is a billing unit, not just a retrieval choice); and meter queries × candidates before you ship, because agent-driven retrieval is quietly turning the cheapest accuracy upgrade in the stack into the line item that grows fastest. Two vendors, two meters, one price band — about $2–2.5 per thousand 100-candidate queries — is as close to a market consensus as AI pricing gets. The corpus is a one-time cost; relevance is a subscription.