What This Check Actually Tells You
AI assistants cannot cite what their crawlers cannot fetch. Before any question of content quality, schema, or entity signals, there is a plumbing question: when OAI-SearchBot or PerplexityBot requests your pages, does your robots.txt turn it away? This tool answers that one question precisely, for the root path of your site, using the same matching logic crawlers use: rules are grouped by user agent, a group that names the bot specifically beats the wildcard group, and within the winning group a root disallow means blocked.
The verdicts carry different weights, which is why every row is labeled by category. A blocked training crawler like GPTBot costs you presence in future model training and nothing else. A blocked AI search crawler like OAI-SearchBot or PerplexityBot removes you from live AI answers today, which for most sites is the single most expensive line in their robots.txt. A blocked user fetcher mostly inconveniences the humans asking about you. And if Googlebot or Bingbot show as blocked, stop reading this page and fix that first, because Search itself is off.
The Accidental Block Pattern
The most common failure we find is not a deliberate policy but an inherited one: a blanket AI block copied from a 2023 blog post, a security plugin that ships aggressive defaults, or a staging robots.txt that went live. The site owner believes they are visible to AI, their content strategy assumes it, and one forgotten Disallow line quietly voids the whole investment. Thirty seconds in this checker settles it.
Reading Your Score
The score weights what matters: access for AI-specific crawlers makes up most of it, an llms.txt adds points because it means the bots that do arrive get a curated map, and having a robots.txt at all earns credit because explicit policy beats accidental defaults. A score in the eighties or above means your access layer is clean and your AI visibility problems, if any, live elsewhere: content, entities, citations. A low score with blocked AI search bots is the rare good news in SEO, a real problem that one file edit genuinely fixes. Our robots.txt builder produces the corrected file, and the complete AI crawler list explains every agent this checker tests, including the compliance records the verdict table cannot capture.
One honest limitation: this reads your published policy, not operator behavior. A bot with a contested compliance record may fetch pages regardless of what your file says, and network-level enforcement is a different layer entirely. What the checker guarantees is that your stated policy and your intended policy match, which in our audit experience is where most of the real-world damage lives. The access layer is one of six signals we score in a full AI visibility audit, alongside schema, entities, content structure, freshness, and actual citation testing across the major engines.
Frequently Asked Questions
The checker says a bot is allowed by default. Is that bad?
It is not bad, it is just unexamined. Allowed by default means no rule in your robots.txt names that agent, so it inherits your wildcard policy. For most sites that is exactly right. The value of an explicit policy is that nobody can change your AI posture by accident when they edit the wildcard rules for something unrelated.
My site has no robots.txt at all. Do I need one?
Without the file, every crawler is allowed everywhere, which is a legitimate maximum-visibility policy. You still want the file for two reasons: to state a Sitemap line, and to make your AI stance deliberate rather than accidental. The builder linked above generates a correct one in a minute.
Does this tool check pages other than my homepage?
The verdict is computed for the root path, which is the policy that governs a site-wide block, the case that matters most. Path-specific rules like a blocked /private/ directory will not change the root verdict, and that is intentional: partial blocks are usually deliberate, root blocks are usually the accident.
A blocked bot keeps showing in my server logs. Why?
Three possibilities: the operator ignores robots.txt, the requests are an impostor wearing the bot's user agent from unrelated infrastructure, or the bot is fetching only your robots.txt itself to check the rules, which compliant crawlers do. Verify the visiting IPs against the operator's published ranges before concluding anything.