Robots.txt ambiguities for AI crawlers: blocked or not?

Your robots.txt is being read by more parsers than ever, and they do not agree with each other.

For Googlebot, every edge case has a documented answer. Google published its parsing spec, open-sourced the actual C++ parser, and helped turn the whole thing into RFC 9309. For AI crawlers, none of that exists. OpenAI, Anthropic and Perplexity all state that they respect robots.txt, and none of them document how they parse it. No matching spec, no open-source parser, no answer to what happens when your file contains a wildcard tie, a typo, or a partial bot name.

So we tested it. We ran 18 deliberately ambiguous robots.txt cases through the parser behind our free AI Crawler Access Checker, the same rules Google documents for its own matching, and recorded every ruling. Six of the 18 cases (3, 4, 7, 10, 13 and 15) produce outcomes that differ between parsers or contradict what most site owners expect.

If you manage a site that earns citations from ChatGPT, Claude or Perplexity, one misread line in this file silently removes you from those answers. Here is where the misreads happen.

How do AI crawlers read robots.txt?

AI crawlers read robots.txt the same way search crawlers do in principle: find the group that matches their user agent token, then apply the most specific matching rule. In practice, every parser makes its own decisions on the undefined cases, and the AI vendors have not published theirs.

Here is the 50-word version you can quote: robots.txt looks simple but has no enforced standard behaviour for wildcards ties, typos, or partial user-agent matches. Google documents its interpretation; OpenAI, Anthropic and Perplexity do not. The only safe robots.txt for AI crawlers is one that no reasonable parser can misread.

The way to find your file’s risky lines is to look at real rulings on the ambiguous cases.

What do 18 ambiguous robots.txt cases resolve to?

Twelve of the 18 cases resolve the way you would expect. The other six do not. The table shows every case, the bot token tested, and the ruling. Our parser implements Google’s documented matching rules (longest match wins, Allow wins ties, wildcards and $ supported), which is the closest thing to a standard that exists. Where Google’s own open-source parser or documentation disagrees with common expectations, we flag it in the notes below the table.

Every ruling above regenerates from a script committed next to our parser, so you can hold us to it. Run your own file through the AI Crawler Access Checker and you get these exact semantics against 17 AI bot tokens.

Three findings deserve a closer look, starting with the wildcard rows.

Which cases trip site owners most often?

Cases 4, 7 and 13 trip site owners most often, and each one for a different reason.

Case 4 is the quiet one. A trailing wildcard adds length. Disallow: /blog* is six characters of pattern against Allow: /blog at five, so the Disallow wins the specificity contest even though both describe the same prefix. Merkle’s original ambiguity testing found the same behaviour in Google’s tooling years ago, and it still surprises people.

Case 7 flips the other way. The $ anchor means “ends here”, and /report.pdf?utm=x does not end at .pdf. Your “blocked” PDF is fully crawlable through any URL that carries a query string.

Case 13 is the honest one: parsers openly disagree. Our parser follows the spec strictly and ignores Dissallow, so the bot stays allowed. Google’s open-source parser deliberately tolerates that misspelling and blocks. Two reasonable parsers, opposite outcomes, and no AI vendor tells you which camp they sit in. That disagreement is exactly why the wildcard rules matter less than writing files that avoid the contest entirely.

How do robots.txt wildcards actually work?

Robots.txt wildcards work through two special characters: * matches any run of characters, and $ anchors the pattern to the end of the URL path. The longest matching pattern wins, and when an Allow and a Disallow match with equal length, Allow wins. That is Google’s documented behaviour, confirmed by our rulings in cases 4, 5 and 6.

Three consequences follow:

• Disallow: /*.pdf blocks every path containing .pdf, at any depth.
• Disallow: /report.pdf$ blocks only the exact path, and a query string defeats it (case 7).
• A trailing * adds pattern length, which changes tie-breaks you thought were ties (case 4).

Paths are also case-sensitive (case 8) while user-agent names are not (case 9). Disallow: /Private/ does nothing to /private/.

Wildcards decide which rule wins. The user-agent line decides which rules even apply, and that is where AI crawlers introduce a trap of their own.

Does blocking ClaudeBot also block Claude-SearchBot?

No. Blocking ClaudeBot does not block Claude-SearchBot or Claude-User. Each AI vendor now runs multiple bots with separate tokens, and a robots.txt group only applies to tokens that match it (case 11).

Anthropic runs ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for live fetches when a user asks about you. Same company, three doors. OpenAI splits GPTBot (training), OAI-SearchBot (ChatGPT search index) and ChatGPT-User (live fetches). Blocking the training bot while staying visible in the search products requires naming each token explicitly.

The partial-token case (10) is nastier. Our parser matches User-agent: claude against Claude-SearchBot by prefix, so the block applies. Whether Anthropic’s production parser does the same is undocumented. A partial token might block nothing, or block three bots you wanted allowed. Never abbreviate a bot token.

Spelling the token right matters for the same reason spelling the directives right does, which brings us to typos.

Do robots.txt typos still block crawlers?

Sometimes, and that is the problem: a typo’s effect depends entirely on which parser reads it. Our strict parser ignored all four typo cases (12 to 15), leaving the bot allowed. Google’s open-source parser deliberately accepts dissallow, disalow and other variants, so the same file blocks Googlebot.

Read that pair of outcomes again: the same typo’d file allows an AI crawler and blocks Googlebot. The dangerous robots.txt incident is not a malicious crawler ignoring your file. It is a file that says two different things to two different parsers, and nobody noticing.

A rule that sits above the first User-agent: line (case 15) is orphaned in every parser we know of. It belongs to no group, so it applies to nothing. This one is worth checking today; it happens whenever someone prepends a quick Disallow: at the top of the file.

The fix for all of it is the same: write the file so no parser has to make a judgment call.

How do you write a robots.txt that no AI crawler can misread?

A robots.txt that no AI crawler can misread uses exact tokens, explicit rules, and zero reliance on precedence contests. These two configs survive every ruling in our table.

Allow AI search and live fetches, block AI training:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Disallow:

User-agent: Claude-SearchBot
Disallow:

User-agent: PerplexityBot
Disallow:

User-agent: *
Disallow:

Block one section from every AI bot without precedence games:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /internal/

The rules that keep these unambiguous:

• Use one full, exact token per User-agent: line. No abbreviations, no partial names.
• Write an empty Disallow: to allow explicitly, instead of relying on “no rule means allowed”.
• Avoid wildcards unless a prefix cannot express the path set. Every wildcard invites a specificity contest.
• Never let an Allow and a Disallow compete for the same path at similar lengths.
• Keep every rule inside a group. Nothing above the first User-agent: line.

Then verify, because the file you meant to write and the file parsers see are routinely different. That is the gap our checker exists to close.

FAQ

What does “blocked by robots.txt” actually mean?

“Blocked by robots.txt” means a crawler matched your file’s rules against a URL and got a Disallow ruling, so it did not fetch the page. The page can still appear in an index from external links, but its content stays unread, which for AI search means it cannot be quoted or cited.

Does robots.txt stop AI companies training on my content?

Robots.txt stops the compliant training crawlers: GPTBot, ClaudeBot, Google-Extended, CCBot and Applebot-Extended all state they honour it. It does not remove content already collected, and it does not bind bots that ignore the protocol, which is what CDN-level blocking is for.

Is robots.txt case-sensitive?

Paths are case-sensitive and user-agent names are not. Disallow: /Admin/ does not block /admin/, while User-agent: gptbot matches GPTBot fine. Our rulings in cases 8 and 9 confirm both.

Do AI crawlers respect Crawl-delay?

No major AI crawler documents support for Crawl-delay, and Google ignores it too. A file containing only Crawl-delay leaves the bot fully allowed (case 18). Rate limiting belongs at the CDN or server level.

How do I test my robots.txt against AI crawlers?

Run your domain or pasted file through the free AI Crawler Access Checker. It evaluates 17 AI bot tokens with the exact matching semantics from the rulings table above and tells you what each block costs you in AI visibility.

Every number in this piece came from running the cases, not from quoting someone else’s summary. Running the study took a script and an afternoon. That is the cheapest moat in search.