Blocking Was Never the New Part
Refusing a crawler has been one line of robots.txt since 1994. That has not changed and it was never difficult. What changed in 2026 is that a third answer became possible, because Cloudflare launched a pay per crawl system that revives HTTP 402 Payment Required, a status code that sat in the specification for about thirty years with nothing behind it.
Until now the answer to may I read you was yes or no, and the only lever a publisher held was to disappear. A price is a third answer, and it turns a refusal into a transaction. That is genuinely new, and it is also the part of the story that matters least to most sites.
The Default Nobody Chose
The more consequential half of the change is the default. New domains on that network start closed to AI crawlers. Nobody was asked, and a refused crawler sends no notification.
If your site is new, check before you write anything else
A default is a decision somebody else made for you and did not mention. Any site launched in the last year that wonders why nothing quotes it should check its crawler policy before rewriting a single page, because no amount of good writing helps a page that nothing can reach.
The Three Lanes at Your Edge
Every machine request now meets one of three answers at your edge. Two of them are a decision and the third is what you get by doing nothing.
| Lane | Status | What you get | What it costs |
|---|---|---|---|
| Open | 200 | Reach, citations, presence in answers | No revenue from the crawl itself |
| Priced | 402 | Revenue only if the operator agrees to pay | Silence from every operator that declines |
| Closed | 403 | Neither reach nor revenue | Everything, and usually by accident |
When Charging Actually Works
A toll only works when refusing costs the other side more than paying. That condition is real for a wire service, a major newspaper, a specialist data provider, anything whose content cannot be substituted. It is false for almost everybody else.
If your content can be replaced by the next result, a crawler that meets a toll routes around it and quotes somebody else. You earn nothing and you lose the visibility. A toll booth on a road nobody has to take is just a closed road.
The substitution test
Ask whether a model answering a question in your category could produce a good answer without you. If yes, you have no pricing power and a toll costs you more than it earns. If genuinely no, you may have a licensing business rather than a content marketing one.
How to Check Your Own Lane
Read your robots.txt as a stranger would
Open it and find every rule that could touch an AI agent, paying closest attention to wildcard rules. A broad rule written years ago for scrapers is usually the real culprit rather than any line naming an agent.
Check your CDN, not just the file
Network level bot rules override anything in your robots.txt and are invisible from the outside. This is where the default lives, and it is the most commonly missed setting in the whole stack.
Request your own page as a crawler
Fetch a live URL with an AI agent user string and read the status code. A 403 tells you the answer immediately. This takes one command and it is the only proof that matters.
Check the logs for what is actually arriving
If an agent you expect to see has never appeared, you are not being crawled by it regardless of what the file says. Absence in a log is the strongest evidence available.
Allowing Search While Refusing Training
The useful position for most businesses is not open or closed, it is per agent. Training crawlers and search index crawlers are different user agents doing different jobs, and they can be given different answers. Refusing training while allowing search keeps you in the answers people see today, which is where the commercial value sits.
If you want to state terms rather than just control access, a machine readable licensing file expresses what an operator may do with content once read. That is a separate layer from robots.txt and it is worth understanding before reaching for a blanket block.
The thing that actually hurts
It is not a policy you picked. It is one you did not know you had. Almost every invisible site we audit is invisible because of a default, not because of a decision.