The Short Answer
To get cited by Claude, let its search and user agents reach your pages, make sure those pages exist in the Brave Search index that Anthropic lists as the provider behind Claude's web search, and publish claims precise enough that a careful assistant can repeat them without hedging: named, dated, specific, and confirmed by pages you do not own. None of it requires a special file, a plugin or a paid placement.
The one sentence version
Claude cites what it can stand behind. Put a checkable fact on a reachable page, get it confirmed somewhere else, and most of the work is done.
Why a Careful Assistant Is Worth the Effort
Anthropic describes honesty and calibrated uncertainty as central to how Claude is trained. In practice that looks like an assistant that qualifies claims, separates what it knows from what it is inferring, and would rather say that a source reports something than state it as settled fact. For a brand, that is a harder audience than a search results page, because a results page will rank you without believing a word you wrote.
The flip side is the reason this guide exists. When an assistant that is built to be careful does name a product, the mention reads as considered rather than promotional, which is exactly the kind of sentence you want a buyer to see. Being named by an assistant that hedges is worth more than being one link among ten on a page that does not.
It also means the usual shortcuts work badly here. Superlatives, unattributed statistics and claims that only your own site makes are precisely the material a careful assistant qualifies or leaves out, so a page written to impress tends to lose to a page written to be checked.
Claude's Three Crawlers
Anthropic documents three separate agents, each with its own user agent string and its own consequence if you block it. Treating them as one bot called Claude is an easy configuration mistake, and an expensive one.
| Agent | What Anthropic says it does | If you block it |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to training its models | Your future pages are excluded from training data |
| Claude-SearchBot | Navigates the web to improve the relevance and accuracy of search responses | Your content is not indexed for Claude's search optimisation |
| Claude-User | Visits pages when a person asks Claude a question | Claude cannot retrieve your page for that person |
According to Anthropic, all three honour robots.txt. That is a real difference from engines whose user triggered fetchers ignore it, and it means your rules for Claude do exactly what they say. Anthropic also supports the non standard crawl delay directive and publishes the addresses its bots use, so a request claiming to be Claude can be checked against a real list instead of trusted on its word.
Why the address list matters
A user agent is a text field that anything can send. On one of the sites we run, a single audit found hundreds of requests wearing AI crawler names while probing for configuration files. Checking against the published address list is the only way to know a request really came from Anthropic.
Where Claude's Web Search Comes From
When Claude searches the web it does not use Google or Bing. Anthropic lists Brave Search as the subprocessor for web search, added in March 2025, and independent testing at the time found Claude's citations matching the results Brave returned for the same queries. Claude then fetches the pages it picks before it writes the answer.
That has a consequence most sites have never considered. Ranking well on Google tells you nothing about whether Brave has indexed you, and a page Brave has never seen cannot be among the candidates Claude chooses from, however good the page is. Brave runs its own independent index rather than renting somebody else's, so it has to find you on its own.
Check whether Brave knows your site
Search Brave for your domain with the site operator, then for your brand name. Missing key pages cannot be surfaced by Claude search.
Request discovery for what is missing
Brave offers a public submit URL form on its search site. It asks Brave to discover or refetch a page and does not guarantee indexing, so treat it as a nudge rather than a fix.
Earn the visits Brave learns from
Brave builds its index partly from anonymous, opt in browsing data contributed by Brave browser users. Pages real people open are easier for it to find than pages nobody visits.
What Our Server Logs Show
We looked at fifteen days of logs across the sites we run on one server, from the 2nd to the 16th of September 2026. Requests identifying as ClaudeBot came to 9,583, Claude-User to 5,845 and Claude-SearchBot to 1,586. Visits that arrived from Claude itself came to 17.
| Agent or signal | 15 days, our sites | What it suggests |
|---|---|---|
| ClaudeBot | 9,583 requests | Training collection is steady and broad |
| Claude-User | 5,845 requests | People are asking Claude questions our pages help answer |
| Claude-SearchBot | 1,586 requests | Anthropic's own search crawling is light, which fits results coming from Brave |
| Visits from Claude | 17 | Claude answers in place and rarely sends the reader on |
Two caveats keep those numbers honest. They are declared user agents counted behind a CDN, not requests verified against Anthropic's address list, so read them as the shape of the behaviour rather than an audit. And the last column is our interpretation of the pattern, not something Anthropic has confirmed. The ratio is still hard to ignore: roughly 340 live fetches for every visit from Claude, against about 2.6 for ChatGPT on the same sites over the same fortnight.
Do not judge Claude by referral traffic
Measure Claude only through analytics and you will conclude it ignores you. The fetch logs say the opposite. Claude reads a great deal and sends very few people on, so the question worth asking is what it says about you, not how many visits it produces.
What Claude Needs Before It Names You
Because a careful assistant qualifies what it cannot support, the practical goal is to remove every reason it would have to hedge about you. That is mostly a writing problem, and it rewards the same discipline as writing sentences a model can lift whole.
A claim it can check
A statement with a number, a date or a named standard gives Claude something to repeat. A phrase like best tool for teams gives it something to qualify or drop.
A source that is not you
A review, a list or a discussion elsewhere that says what your page says gives a careful assistant something to stand on. Corroboration turns a claim into a finding.
A date on every fact
An undated figure forces a careful reader to wonder whether it is still true. A dated one can be repeated with confidence, or with the date it was last true.
Nothing inflated to trip over
One exaggerated claim invites caution about everything around it. Pages that state limits plainly, including what the product does not do, read as trustworthy to careful readers of every kind.
Notice that none of those four is specific to Claude. They are simply the properties that matter most to the reader least willing to take your word for things, which is why fixing them for Claude tends to improve how every other assistant describes you as well.
Robots.txt Settings That Matter
Most sites that are invisible to Claude never decided to be. A broad rule written years ago for scrapers, or a CDN setting that blocks AI crawlers by default, removes Claude-SearchBot and Claude-User along with everything else. The fix is to name the agents you want explicitly, the same principle behind a sensible AI crawler allow list.
| Goal | ClaudeBot | Claude-SearchBot | Claude-User |
|---|---|---|---|
| Citable, and allow training | Allow | Allow | Allow |
| Citable, opt out of training | Disallow | Allow | Allow |
| Invisible to Claude entirely | Disallow | Disallow | Disallow |
The middle row is the one most publishers actually want and very few configure. It keeps your pages available to Claude's search and live answers while telling Anthropic not to use future pages for training, and because all three agents honour robots.txt, the rule does what it says.
How to Test Whether Claude Cites You
Start a fresh conversation with web search on
Earlier chats and project context colour the answer. Start clean every single time.
Ask a category question without your brand name
Ask which tools people use for the job your product does. Naming yourself only tests recognition, not recommendation.
Run it three times and record every name
Answers vary from run to run. Three runs show whether an appearance is consistent or luck.
Repeat with web search off
This shows what the model remembers without retrieval. Absent here but present with search on means the live path is carrying you.
Read the qualifiers, not just the names
Note whether Claude states your facts plainly or hedges them. Hedging on a correct fact usually means it found too little confirmation.
Repeat the test monthly with identical wording and compare it with how other assistants answer the same question. Each engine needs tracking on its own terms, and Claude will rarely match ChatGPT or Perplexity name for name, because it searches a different index and weighs claims differently.
When Web Search Is Off
Claude does not search the web for every question. When it answers from what it already knows, the only way you are in that answer is through training data, which is the job ClaudeBot does. That is why the ClaudeBot decision is a real trade rather than a formality: blocking it keeps future pages out of training, and with them out of every answer Claude gives without searching.
There is no dashboard for what a model remembers. The only test is the one above with web search switched off, repeated over months, because memory changes only when a newer model is trained, on a schedule Anthropic sets and you cannot see. Most SEO reporting never looks at Brave or at model memory at all, so gaps in either can go unnoticed for as long as nobody checks.
Mistakes That Keep Sites Out of Claude
Blocking by wildcard
A rule written for scrapers that also removes Claude-SearchBot and Claude-User without anybody noticing.
Trusting the name in the header
Allowing or blocking traffic based on a user agent that any script can fake in one line.
Optimising for Google alone
Assuming a strong Google position means Brave, and therefore Claude search, can find the page too.
Writing to impress instead of to be checked
Superlatives and unsourced numbers that a careful assistant qualifies, hedges or quietly leaves out.
The Checklist
Allow Claude-SearchBot and Claude-User
Name both explicitly in robots.txt, and decide about ClaudeBot as a separate question.
Confirm Brave has indexed your key pages
Check with a site search on Brave and request discovery for anything missing.
Verify requests by address, not by name
Check suspicious traffic against the address list Anthropic publishes.
State facts precisely, with dates
Numbers, standards and limits, each with the date it was last true.
Earn corroboration off your own site
Reviews, lists and discussions that repeat your claims in words that are not yours.
Measure answers, not referrals
Test monthly in fresh conversations and track the wording as well as the mention.
The Bottom Line
Claude reaches the web through different doors than the assistants most guides are written for, and it is more careful about what it will say once it arrives. Open the doors it actually uses, get into the index it actually searches, and give it facts it can stand behind. The reward is a mention from the assistant least inclined to hand one out, which is exactly what makes it worth having. For the same diagnosis applied to the most used assistant, see why a site does not show up in ChatGPT.