Home › Glossary

What is AI crawler blocking (and which bots should your site actually allow)?

Plain-English definition · Updated 2026-11-03. Every rule below is quoted from OpenAI's, Anthropic's and Google's own crawler documentation and Cloudflare's announcements; the blocking-decision framework is ours.

AITop is an independent guide. Prices and features change fast — always check the vendor's page.

The 30-second answer

AI crawler blocking means deciding which automated AI agents may fetch your site — through robots.txt tokens first, and increasingly through network-level defaults. It is not one switch, because crawlers do three different jobs: training data (OpenAI's GPTBot, Anthropic's ClaudeBot), search indexing (OAI-SearchBot, Claude-SearchBot), and user-initiated fetches (ChatGPT-User, Claude-User). OpenAI's docs state the settings are independent — "a webmaster can allow OAI-SearchBot… while disallowing GPTBot to indicate that crawled content should not be used for training" — and Google's Google-Extended token "does not affect a site's inclusion or ranking in Google Search." Blocking the training bot is close to free today; blocking the search bot removes you from AI answers that cite sources; and since 1 July 2025 Cloudflare blocks AI crawlers by default on new domains, so some sites are opted out without having written a single robots.txt line.

What it actually means

The mechanism is a text file. robots.txt maps user-agent tokens to allow/disallow paths, and each AI vendor now documents its own tokens. OpenAI documents four bots in its crawler guide: GPTBot, which crawls content "that may be used to train OpenAI's generative AI foundation models"; OAI-SearchBot, which fetches pages "to show in ChatGPT search results" and publishes its IP ranges at openai.com/searchbot.json; ChatGPT-User, used "when a user asks ChatGPT to visit a website" — and because these actions are user-initiated, OpenAI notes that "robots.txt rules may not apply" to it; and OAI-AdsBot, which verifies pages submitted as ads, with the explicit statement that its data "is not used to train generative AI foundation models." Two operational details matter: if a site allows both search and training bots, OpenAI "may use the results from just one crawl for both use cases," and after you edit robots.txt "it can take ~24 hours from a site's robots.txt update for our systems to adjust."

Anthropic documents three bots in its help center: ClaudeBot collects "web content that could potentially contribute to training"; Claude-User retrieves pages "when individuals ask questions to Claude"; Claude-SearchBot navigates "to improve search result quality." All three honor robots.txt and the (non-standard) Crawl-delay directive. One warning in Anthropic's docs is easy to miss and worth quoting in full: blocking the bots' IP addresses "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." In other words, an IP firewall is not a stricter robots.txt — it can permanently break the polite opt-out channel.

Google is the special case. Google-Extended is described in Google's crawler documentation as "a separate product token" — and, unlike every bot above, it "does not have its own HTTP request user-agent string; crawling is done with the existing Google user-agent strings." It controls whether Google-crawled content may be used "to train future versions of Gemini models" and to ground Gemini answers, and Google states that using it "does not affect a site's inclusion or ranking in Google Search, nor is it used as a ranking signal." You cannot split Googlebot's search indexing from Google's AI data collection with robots.txt alone — the token governs downstream use, not the crawl.

Why it matters when you're picking a tool

Blocking is per-purpose, and the consequences are wildly asymmetric. Disallow a training bot and today's visible cost is roughly zero — your pages still rank in classic search, still load, still convert. Disallow a search bot and the cost is immediate: OpenAI's docs state that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links," and Anthropic warns that disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results." With AI answers now displacing clicks, that asymmetry is the whole decision. The scale of it is measurable: per Cloudflare's 2025 data, the crawl-to-referral ratio was roughly 14:1 for Google Search, about 1,700:1 for OpenAI and about 73,000:1 for Anthropic — crawls per referral click sent back.

robots.txt is voluntary, and the enforcement layer moved in 2025. In September 2024 Cloudflare shipped a one-click AI-crawler block that more than one million customers enabled. On 1 July 2025 it flipped the default: "every new domain will now be asked if they want to allow AI crawlers," making Cloudflare "the first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default" — with a Pay Per Crawl beta that charges per fetch using the HTTP 402 status code. Cloudflare fronts roughly 20% of web traffic by its own account, so for a growing share of the web the robots.txt gate sits behind a network gate that may already be shut. Any site owner whose analytics show zero AI-bot traffic should check the host-level switch before concluding their content is simply uninteresting to crawlers.

The reverse risk is real too: some crawlers are not who they claim to be. A robots.txt line is a request, an IP allowlist is a check, and identity fraud between the two is a documented failure mode — in August 2025 Cloudflare publicly accused Perplexity of using stealth, undeclared crawlers that rotated user-agents and networks to reach blocked sites (Perplexity disputed the findings). The practical takeaway for a buyer is to treat bot traffic the way you treat any other vendor claim: verify against published IP ranges where the vendor provides them (OpenAI publishes per-bot JSON lists; Anthropic publishes its own IP list), rather than trusting a user-agent string in your logs.

For tool buyers rather than site owners, the same gate works in your favor. Every comparison on this site assumes the underlying model saw enough good data and that AI search can cite real sources — behaviors that only exist because someone, somewhere, allowed a crawler in. When you evaluate an AI search tool, its crawl policy is a quality signal: tools that honor robots.txt and publish their bots (see the vendor pages quoted above) can be opted out of by any site; tools that crawl covertly remove that control from everyone whose content they index — including the sites behind results you're being shown.

The 2026 reality check

The default flipped from opt-out to opt-in. For thirty years, webmasters who did nothing were crawled by everyone. Since mid-2025, a new Cloudflare-fronted domain that does nothing blocks AI crawlers. That means the "do I block AI bots?" question is no longer only asked by publishers with legal departments — it is silently answered for millions of sites at signup, and reopening the gate is now the deliberate act. If your AI-search referrals vanished in 2025-2026, audit the infrastructure layer first.

Per-subdomain still matters. Anthropic's instructions ask owners to set robots.txt "for every subdomain that you wish to opt out from," and OpenAI's 24-hour adjustment window means a launch-week robots.txt mistake is visible for roughly a day before it is fixed. Editors of large sites should also remember the mixed-crawler problem: Google-Extended does not stop Googlebot from crawling, only from feeding Gemini training and grounding — so "we blocked AI training" and "Google stopped crawling us" are different claims, and only the second one would actually hurt your search traffic.

Where this leaves an AI-tool buyer: the crawler gate is becoming a negotiation between three parties — your robots.txt, your infrastructure provider's defaults, and the AI vendors' published policies. Vendors are starting to pay for the crawl (Cloudflare's Pay Per Crawl, TollBit-style middlemen) precisely because the old currency, referral traffic, collapsed to 1,700:1 and 73,000:1 ratios. None of that changes which AI chatbot or writing tool you should buy — but it explains why AI answers cite fewer sources than search results, and why a site that wants to be cited (like this one, which keeps its robots.txt open to search bots and publishes an llms.txt) has to make that choice explicitly.

Quick checklist

Where you'll hit it

Every AI-search comparison on this site is downstream of a crawl decision. Our Perplexity vs Google AI Mode vs ChatGPT Search page compares tools that answer only as well as the pages they were allowed to index; our AI agent entry covers the third job — user-initiated fetches, where OpenAI itself says robots.txt "may not apply." Related terms: RAG (retrieval-augmented generation) is why citation quality tracks crawl policy, and open weights is the other end of the same "who gets your content" question. Full list on the AI tools glossary.

The bottom line

AI crawler blocking is not one decision; it is three. Training bots are the ones everyone argues about and nearly free to block. Search bots are the ones worth reading before you block, because they decide whether AI answers cite you at all. User-action bots sit in between and are barely governed by robots.txt anyway. Layer on the 2025 shift — Cloudflare blocking AI crawlers by default for new domains, pay-per-crawl billing on the way — and the default posture of the web has quietly reversed from "crawl unless forbidden" to "forbidden unless allowed." Make your choice explicit: a robots.txt you wrote on purpose beats one that was written for you at signup. Crawl-to-referral arithmetic (14:1 for search, 73,000:1 for one AI vendor, per Cloudflare's 2025 data) is the number that explains why the negotiation started.