Why AI Crawler Policy Is Now a Business Decision
In 2023, deciding which bots could access your site was a technical chore. In 2026, it decides whether your brand exists inside ChatGPT, Claude, Perplexity, Gemini, and Copilot. The stakes moved from server load to market presence, and most sites are running a robots.txt written before any of these bots existed.
The scale is not subtle anymore. Cloudflare's network data put AI crawlers at roughly 20 percent of all verified bot traffic by mid 2026, with AI search bots adding several points on top. In our own portfolio logs, AI crawlers and fetchers routinely outnumber Googlebot requests on content-heavy sites. Whatever policy you choose, you are already serving these bots or already refusing them. The only question is whether the choice is deliberate.
The economics are lopsided, and you should understand that before choosing a policy. Industry measurements in 2026 put OpenAI's crawl-to-referral ratio at over a thousand pages crawled for every visitor sent back. Training crawlers consume far more than they return in clicks. What they return instead is presence: when a model trains on your content or an AI search index includes it, your brand becomes part of the answer layer where buying decisions increasingly start. That trade, crawl load and content reuse in exchange for AI visibility, is the whole game. This guide gives you the facts to price it.
The Four Kinds of AI Bots
Almost every bad robots.txt decision we audit comes from treating AI bots as one category. There are four, and they deserve four different answers.
The One-Sentence Version
Training crawlers feed tomorrow's models, search crawlers feed today's answers, user fetchers act for a live human, and the Extended tokens are permission switches, not bots. Decide per category, never with one blanket rule.
The Master Crawler Table
The reference table. Every user agent below is the exact string you match in robots.txt and in your server logs. Type is the category from above. The last column reflects each operator's stated policy and its observed track record as of July 2026, which are not always the same thing.
| User Agent | Operator | Type | Feeds | Respects robots.txt |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | GPT model training data | Yes |
| OAI-SearchBot | OpenAI | Search index | ChatGPT search results and citations | Yes |
| ChatGPT-User | OpenAI | User fetcher | Live answers when a user browses | Partially, acts for the user |
| ClaudeBot | Anthropic | Training | Claude model training data | Yes |
| Claude-SearchBot | Anthropic | Search index | Claude web search results | Yes |
| Claude-User | Anthropic | User fetcher | Live answers when a user asks | Partially, acts for the user |
| Googlebot | Search index | Google Search and AI Overviews | Yes | |
| Google-Extended | Control token | Gemini and Vertex AI training permission | Token, never fetches | |
| GoogleOther | Training | Google research and development crawls | Yes | |
| Google-CloudVertexBot | Search index | Vertex AI Search for cloud customers | Yes | |
| PerplexityBot | Perplexity | Search index | Perplexity answers and citations | Disputed record |
| Perplexity-User | Perplexity | User fetcher | Live page visits inside answers | States it may not |
| Applebot | Apple | Search index | Siri, Spotlight, Safari suggestions | Yes |
| Applebot-Extended | Apple | Control token | Apple foundation model training permission | Token, never fetches |
| Meta-ExternalAgent | Meta | Training | Llama and Meta AI training data | Yes |
| Meta-ExternalFetcher | Meta | User fetcher | Link previews and live fetches in Meta AI | Partially, acts for the user |
| Amazonbot | Amazon | Search index | Alexa answers and Amazon AI features | Yes |
| Bingbot | Microsoft | Search index | Bing Search and Copilot answers | Yes |
| Bytespider | ByteDance | Training | Doubao and ByteDance model training | Historically poor |
| CCBot | Common Crawl | Training | Open datasets used by many AI labs | Yes |
| cohere-training-data-crawler | Cohere | Training | Cohere model training data | Yes |
| MistralAI-User | Mistral | User fetcher | Le Chat live answers | Partially, acts for the user |
| DuckAssistBot | DuckDuckGo | Search index | DuckAssist AI answers | Yes |
| GrokBot | xAI | Training | Grok model training data | Yes |
Beyond these, a long tail of smaller operators crawls with declared agents: YouBot for You.com, ExaBot for Exa, Diffbot for structured extraction, PetalBot for Huawei, Timpibot for Timpi, DeepSeekBot for DeepSeek, AI2Bot for the Allen Institute, and retrieval fetchers like FirecrawlAgent, TavilyBot, and LinkupBot that power other companies' AI products. Individually each is a rounding error in your logs. Collectively they matter, which is why our recipes below handle them with a default rule rather than sixty individual entries.
OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
OpenAI runs the cleanest three-way split in the industry, one agent per category, and publishes IP ranges for each so you can verify what claims to be OpenAI actually is.
GPTBotcollects training data for future GPT models. It is the most blocked AI crawler on the web. Cloudflare's robots.txt analysis in early 2026 found it named in more disallow rules than any other AI agent. Blocking it keeps your content out of future training runs and nothing else. Your visibility inside ChatGPT's live product does not depend on it.
OAI-SearchBot builds the index behind ChatGPT search. This is the one that determines whether ChatGPT can find, cite, and link your pages when hundreds of millions of users ask questions in your category. If AI citations matter to your business, this agent stays allowed. Blocking OAI-SearchBot while investing in ChatGPT citation optimization is paying for a billboard and then wrapping it in a tarp.
ChatGPT-Userfires when a user asks ChatGPT to open a page. It is a human's request wearing a bot's user agent. These visits can convert, because someone specifically asked about your page, so blocking this agent turns away warm traffic to protect nothing.
Anthropic: ClaudeBot, Claude-SearchBot, Claude-User
Anthropic mirrors the same three-way structure. ClaudeBot gathers training data, Claude-SearchBot feeds Claude's web search, and Claude-User fetches pages a person asked about. Older strings you may still see in guides, anthropic-ai and Claude-Web, are legacy names; current logs show the three agents above.
ClaudeBot earned a reputation for aggressive crawl rates in 2024, and it has been climbing again: network measurements in June 2026 showed ClaudeBot's share of AI crawl traffic surging to second place among all AI bots. Two practical notes from our own logs. First, ClaudeBot responds well to Crawl-delay directives, so rate problems are solvable without blocking. Second, Anthropic publishes no verification IP list; it treats robots.txt as the control surface. Anything claiming to be ClaudeBot from a data center you cannot reverse-resolve deserves suspicion.
Google, Apple, and the Token Trap
Google and Apple both split crawling from permission, and misreading that split is the most expensive robots.txt mistake on this page.
Google fetches with Googlebot, the same agent it has always used, and that single crawl feeds Search, AI Overviews, and AI Mode. You cannot block Google's AI features while keeping Google rankings, because it is one crawl. What you can control is training: the Google-Extended token governs whether your content trains Gemini and Vertex AI models. Disallow Google-Extended and you exit future Gemini training runs while your rankings and AI Overview eligibility stay intact. Apple works the same way: Applebot crawls for Siri and Spotlight, and the Applebot-Extended token governs foundation model training.
The Mistake We Keep Finding in Audits
Sites that blocked Googlebot or Applebot intending to block AI training, and sites that blocked Google-Extended believing it removes them from AI Overviews. Both wrong, in opposite directions. Blocking Googlebot removes you from Search itself. Blocking Google-Extended does not touch AI Overviews. If you take one technical fact from this page, take that one.
Perplexity and the Trust Problem
Perplexity documents two agents, PerplexityBot for its index and Perplexity-User for live fetches, and its answers cite sources prominently, which makes it one of the better citation traffic sources in the AI search category. It is also the operator with the most contested compliance record on this list.
In August 2025, Cloudflare published evidence that when Perplexity's declared crawlers were blocked, requests continued from undeclared agents impersonating a desktop Chrome browser, rotating IPs and networks to evade blocks, at a scale of millions of requests per day. Cloudflare removed Perplexity from its verified bots program. Perplexity disputed the findings and called the report a publicity stunt. We are not the referee in that dispute, but the practical takeaway is simple: treat robots.txt as a preference signal for Perplexity, not an enforcement mechanism. If you genuinely need Perplexity kept out, that requires network-level blocking. And note the asymmetry of its own documentation: Perplexity states plainly that Perplexity-User may ignore robots.txt because it acts on a user's behalf.
For most brands we advise the opposite of blocking. Perplexity's citation behavior sends real referral traffic, its users skew toward researchers and buyers, and the optimization playbook for Perplexity is one of the highest-yield efforts in GEO right now.
Meta, Amazon, Microsoft, ByteDance, Common Crawl
Metaconsolidated its crawling into Meta-ExternalAgent for training and index building, with Meta-ExternalFetcher handling user-triggered fetches for Meta AI. The older FacebookBot still appears for platform features. Meta AI's enormous user base makes this training pipeline one of the larger visibility surfaces you can opt into or out of.
Amazon crawls with Amazonbot, feeding Alexa answers and Amazon's AI shopping features. For product businesses this one is underrated: Amazon's Rufus assistant answers shopping questions for a buyer base that is already holding a credit card. Ecommerce brands blocking Amazonbot are usually doing it by accident, inherited from a blanket rule. If that is you, read our guide to GEO for ecommerce before deciding.
Microsoft needs no new agent at all. Bingbot feeds both Bing and Copilot, the same single-crawl pattern as Google, which means blocking Bingbot to avoid Copilot also removes you from Bing Search. There is no Copilot-only switch.
ByteDanceruns Bytespider for model training. For years it was the canonical example of a crawler that ignored robots.txt, and it remains widely blocked for that reason. Network data in mid 2026 showed its activity pattern shifting, with volumes reversing after a long decline, so watch your logs rather than assuming yesterday's behavior.
Common Crawl is the quiet one that matters more than its traffic suggests. CCBot builds the open dataset that countless AI labs, startups, and research models train on. Blocking CCBot removes you from a whole generation of models you have never heard of yet. Allowing it is the single broadest training opt-in that exists.
The Decision Framework
There is no universally correct policy. There is a correct policy for your business model. Work through these steps in order:
Identify your revenue relationship with content. If content attracts customers to a service or product, AI visibility is marketing and you want maximum crawler access. If content is the product itself, subscriptions, licensing, paywalled journalism, then training crawlers consume your inventory and deserve scrutiny.
Separate the four categories before writing a single rule. Decide training crawlers, search crawlers, user fetchers, and control tokens independently. A blanket block or a blanket allow is almost always leaving value on the table in one direction.
Default search crawlers to allowed. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot, and Bingbot are where citations come from. Blocking them is exiting AI search the way noindex is exiting Google. Very few businesses genuinely want that.
Leave user fetchers alone. ChatGPT-User and its peers are humans asking about you. They can also reach your pages regardless of robots.txt in many cases, so a block here costs goodwill traffic and enforces nothing.
Make the training call deliberately. Blocking GPTBot, ClaudeBot, Meta-ExternalAgent, and CCBot while allowing everything else is a coherent conservative position. So is allowing all of them for maximum model presence. Pick one on purpose and write down why.
Set the tokens to match the training call. Whatever you decided for training crawlers, mirror it in Google-Extended and Applebot-Extended, since they govern the same question inside Google's and Apple's pipelines.
Check your logs quarterly. Operators add agents, change behavior, and occasionally earn or lose trust. A policy set in January and never revisited is a policy someone else is setting for you.
Three robots.txt Recipes
Three complete, copy-paste policies. Every agent name is exact. Adjust paths to your site and keep your existing sitemap line.
Recipe 1: Full AI visibility. The policy we run on our own sites and most client sites. Everything is allowed, training included, because model presence is marketing. This condensed version covers the majors; the principle extends to any declared agent.
User-agent: * Allow: / # Explicit welcome for the major AI agents User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Amazonbot Allow: / User-agent: CCBot Allow: / Sitemap: https://yourdomain.com/sitemap.xml
Recipe 2: Visible in AI search, out of training.The split position for content businesses. AI assistants can find and cite you today, but your content does not train tomorrow's models. Note the Extended tokens doing the training opt-out for Google and Apple while Googlebot and Applebot stay untouched.
User-agent: * Allow: / # Block training crawlers User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: GoogleOther Disallow: / # Training permission tokens (not crawlers) User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / # Keep AI search and user fetchers open User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / Sitemap: https://yourdomain.com/sitemap.xml
Recipe 3: Full AI lockout. For sites whose content is the product and whose lawyers have opinions. Understand what this buys and what it does not: compliant operators will stay out, user fetchers may still arrive when a human asks, and a robots.txt line does nothing against an operator that ignores the file. A real lockout pairs this with network-level enforcement at your CDN or firewall.
# Block all major AI agents, keep classic search User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: Claude-User Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: * Allow: / Sitemap: https://yourdomain.com/sitemap.xml
One habit worth adopting alongside any recipe: pair robots.txt with an llms.txt file so the AI systems you do allow get a clean map of what matters on your site. We covered that standard, including what it can and cannot do, in our complete llms.txt guide.
Verifying Real Bots vs. Impostors
User agent strings are plain text. Any scraper can claim to be GPTBot, and plenty do, because sites treat the big names generously. Three verification layers, in increasing order of effort:
While you are in the logs, measure the other direction too: which AI agents visit, how often, and which pages they favor. That data tells you whether your GEO work is landing before a single citation shows up. Our guide to tracking AI citations covers the full measurement stack, and the crawler accessibility items in our technical SEO checklist make sure nothing on your side is silently turning these bots away.
Frequently Asked Questions
Should I block GPTBot?
For most businesses, no. GPTBot only feeds model training, so blocking it does not remove you from ChatGPT search, which is governed by OAI-SearchBot. If your content markets a product or service, training presence is free brand distribution. Blocking GPTBot makes sense mainly when content is the product itself and licensing is the goal.
What is the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?
GPTBot collects training data for future OpenAI models. OAI-SearchBot builds the live index that ChatGPT search uses to find and cite pages. ChatGPT-User fetches a specific page in real time because a user asked ChatGPT to open it. They can be allowed or blocked independently in robots.txt, and the visibility cost of blocking each is very different.
Does blocking AI crawlers hurt my Google rankings?
Blocking dedicated AI agents like GPTBot or ClaudeBot has no effect on Google rankings. The danger is Googlebot itself: Google Search, AI Overviews, and AI Mode all run on the Googlebot crawl, so blocking it removes you from everything. The Google-Extended token is the safe lever, since it only opts you out of Gemini training and leaves Search and AI Overviews untouched.
How do I know if a crawler claiming to be GPTBot is real?
Check the visiting IP, not the user agent string. OpenAI publishes JSON files of official crawler IP ranges, so a real GPTBot request comes from a listed range and a fake one does not. For operators without published ranges, run a reverse DNS lookup on the IP and confirm it resolves to the operator's domain. Anything from an anonymous data center wearing a famous name is a scraper.
How much of my traffic is AI crawlers?
More than most owners expect. Network-level measurements in 2026 attribute roughly a fifth of verified bot traffic to AI crawlers, and on content-heavy sites in our own portfolio, AI agents regularly generate more requests than Googlebot. Your server logs or CDN dashboard can give you the exact figure within an hour of looking.
Crawler policy is the plumbing of AI visibility: invisible when right, expensive when wrong. If you want a second pair of eyes on your configuration, or a full picture of how AI systems currently see your brand, our AI visibility audit covers crawler access, entity signals, and citation performance in one pass.
Related
LLMs.txt: The Complete Guide to What It Does (and Doesn't Do)
Read guide
Related
Technical SEO Checklist 2026
Read checklist
Abd manages crawler access and log analysis across the Outline portfolio and client sites, and keeps this reference current as operators add agents and change behavior. Corrections and newly observed agents are folded in with each revision.