Why AI Bots Need Four Different Answers
The most expensive robots.txt mistakes we find in audits come from one habit: treating every AI bot as the same thing. A training crawler consuming your archive for a future model, a search crawler indexing you for tomorrow's ChatGPT answers, a fetcher retrieving one page because a live human asked, and a permission token that never fetches anything are four different deals with four different costs. Block them all and you have exited AI search to stop training that a token could have stopped for free. Allow them all without deciding and someone else made your content policy for you.
The presets encode the three positions that actually make sense. Full visibility treats model presence as marketing, which is the right call for most businesses whose content exists to attract customers. The split keeps you citable in AI search while opting out of training runs, the coherent conservative position for content businesses. Lockout is for sites whose content is the product itself, and it comes with a warning the tool bakes into your thinking rather than your file: robots.txt is a request, not a wall, so a true lockout also needs enforcement at your CDN.
The Two Rules People Get Wrong
Never block Googlebot or Bingbot to stop AI: Google Search, AI Overviews, Bing, and Copilot all ride those two crawls, so blocking them removes you from search itself. And do not expect Google-Extended to remove you from AI Overviews: it only governs Gemini training. Both mistakes come from the same confusion between crawlers and permissions, which is exactly why those two agents live in their own category above.
Installing Your File Safely
robots.txt lives at your site root and there is exactly one of it, so the file this tool produces replaces your current one rather than adding to it. Before uploading, carry over anything site-specific you already rely on: Disallow rules for admin paths, crawl-delay lines, extra sitemaps. The generated file deliberately starts from a permissive default for everything that is not an AI agent, because a broken robots.txt fails silently and the damage shows up weeks later as vanished rankings. After uploading, confirm yoursite.com/robots.txt serves the new version, then run it through our AI crawler checker, which reads your live file back the way the bots do and tells you whether reality matches your intent.
If you are deciding what your policy should even be, the full reasoning lives in our complete AI crawler list: every agent in this builder with its operator, compliance record, and what blocking it actually costs. Pair the finished policy with an llms.txt file so the bots you allowed also get a curated map of what matters.
Frequently Asked Questions
Will blocking AI crawlers hurt my Google rankings?
Blocking dedicated AI agents like GPTBot or ClaudeBot has zero effect on Google rankings. The only dangerous blocks are Googlebot and Bingbot themselves, which is why this builder never touches them. The Google-Extended token controls Gemini training without affecting Search or AI Overviews.
Do AI companies actually respect these rules?
The major operators state that their training and search crawlers honor robots.txt, and for OpenAI, Anthropic, Google, and Apple the observed record broadly matches. User-triggered fetchers are the documented exception, since they act on a human's behalf. For operators with contested records, robots.txt is a preference signal and real enforcement happens at the network level.
Which preset should I pick?
If your content markets a product or service, full visibility: being inside AI answers is distribution you would otherwise pay for. If your content is the product, the split preset keeps you citable while staying out of training. Lockout only makes sense with CDN enforcement behind it and a licensing strategy in front of it.
Why are Google-Extended and Applebot-Extended listed if they are not crawlers?
Because they only work in robots.txt. They are permission switches that Googlebot and Applebot check to learn whether already-crawled content may train Gemini and Apple's models. Leaving them out of an AI policy silently opts you into training you may have meant to refuse.