A B2B software company added one wildcard line to robots.txt last year: block AI, all of it. Six months later, ChatGPT was still quoting their pricing page almost word for word. Claude never mentioned them. Not once.
The content hadn’t changed. What changed was the assumption underneath the rule, that “AI” is a single category a single line can stop. It isn’t. ClaudeBot and the half-dozen other named crawlers roaming the web each read robots.txt on their own terms, and a generic rule written for all of them ends up working for none of them.
Key Takeaways
- ClaudeBot is its own user agent, distinct from Anthropic’s other crawlers. One “disallow all AI” line won’t reliably catch it, or let it through.
- Anthropic’s declared training crawler has been documented, and reported by site owners, as following robots.txt more consistently than some of the crawlers Perplexity uses for live, on-demand fetches.
- Training-data crawls and real-time retrieval come from different bot types entirely. Blocking by exact agent name works. Blocking by brand name doesn’t.
There’s No Such Thing As “The AI Crawler”
Every lab runs more than one bot, and each one has a job. OpenAI has GPTBot for training, ChatGPT-User for when someone asks ChatGPT to open a specific link mid-conversation, and OAI-SearchBot for its search index. Anthropic runs ClaudeBot as its main crawler, kept separate from whatever fetches a page live because a user typed a question about it.
Write “Disallow: GPT” or “Disallow: AI” into robots.txt and the parser matches exactly nothing. These bots announce themselves with specific strings: ClaudeBot, GPTBot, PerplexityBot. If the goal is to let Claude in while keeping competitors’ scrapers out, or the other way around, the rule has to name the agent. Vague patterns don’t average out to “sort of blocked.” They either block everything or they block nothing, depending on how the parser handles the wildcard.
What ClaudeBot Actually Does When It Hits Your Site
ClaudeBot sends that exact string in its user-agent header, no variation. Anthropic’s own documentation says it checks robots.txt before every fetch and honors Allow and Disallow rules scoped to its block. One caveat: it doesn’t reliably respect a nonstandard Crawl-delay directive, so don’t lean on robots.txt alone for rate limiting.
Policy pages are one thing. Logs are another.
Grep your access logs for “ClaudeBot” across a few weeks. Compare request paths against what robots.txt claims to allow. If ClaudeBot is hitting pages marked disallowed, the crawler probably isn’t the problem, something upstream is. A CDN rule. A cache serving stale permissions. A bot-management layer quietly overriding the file you wrote.
Writing the Rule, Agent by Agent
Give each crawler its own user-agent block. Don’t lump them under one wildcard section. A site that wants Claude’s training crawler in, but wants a scraper kept out, writes something close to this:
User-agent: ClaudeBot
Allow: /
Disallow: /internal/
Disallow: /account/
User-agent: GPTBot
Disallow: /
User-agent: PerplexityBot
Allow: /
Specificity wins here, not order on the page. Most parsers, including the ones these labs run, apply the most specific matching block for an agent rather than merging it with a wildcard (User-agent: *) section further down. A wildcard Disallow sitting above these named blocks won’t silently swallow them. But it also won’t protect any path the named block explicitly allows. Run each block through a parser and check it. Robots.txt doesn’t execute top to bottom like a script, and treating it that way is how teams end up with rules that look right and do nothing.
Getting cited starts with a model finding the page at all. Once it starts citing a brand consistently, that habit tends to compound.
total AI citations and number 1 in category for Moburst’s own brand, across 309 unique pages cited.See the case study
Why Claude’s Behavior Isn’t Perplexity’s Behavior
Anthropic says, in public documentation, that ClaudeBot checks robots.txt before every fetch. That’s a vendor claiming good behavior, sure. But it lines up with what publishers and bot-management vendors have found auditing their own logs: ClaudeBot requests tend to stop cold at a disallowed path. They don’t come back after a Disallow goes up.
Perplexity is a different story. Publishers and researchers have reported Perplexity fetching and summarizing pages that explicitly blocked its declared crawler. Perplexity’s response has been to point at a separate, user-triggered fetch mechanism, distinct from the named crawling agent. That distinction is the whole problem in miniature: a crawler that respects robots.txt and a fetcher that grabs one page because a user asked a question aren’t the same kind of thing, even under the same company logo.
The crawler that trains on your content and the crawler that fetches it on demand are not the same thing, even when they share a brand name.
ChatGPT sits in between. GPTBot, OpenAI’s training crawler, has generally been reported as compliant with rules scoped to its name. ChatGPT-User, the agent that fires when someone asks ChatGPT to open a link during a live chat, acts more like a browser following a direct human instruction. That’s a different trust relationship than a background training crawl. Treat it like one.
What To Actually Do This Week
Pull thirty days of server logs. Search for each named agent separately: ClaudeBot, GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User. Write down which ones are really requesting pages, then compare that against what your robots.txt file assumes is happening. These two lists almost never match on the first try.
Rewrite robots.txt with one block per agent instead of a single wildcard section, using the pattern above as a template. Then check whether a CDN, firewall, or bot-management tool is layering its own block list on top. That layer overrides the file completely, and it’s the single most common reason a technically correct robots.txt still produces crawler behavior nobody expected.
FAQs
Does “Disallow: /” Under User-Agent: * Block ClaudeBot?
Only if there’s no separate User-agent: ClaudeBot block anywhere in the file. Most parsers, Anthropic’s included, apply the most specific matching block for an agent rather than folding it into the wildcard section. A named ClaudeBot block with its own Allow directive beats a wildcard Disallow every time.
Does Blocking ClaudeBot Keep a Site Out of Claude’s Answers?
Not entirely. Blocking the crawler stops future training data collection. It doesn’t erase content already absorbed from earlier crawls, and it has no effect on answers Claude builds by reasoning over publicly known facts rather than quoting a specific page.
Why Did ChatGPT Keep Citing a Page We Thought We Blocked?
Check which agent actually made the request. A rule aimed at GPTBot does nothing to ChatGPT-User, which fires when someone asks ChatGPT to open a link during a live conversation. It behaves more like a direct fetch than a background crawl, and it’s governed by different rules.
How Often Should These Rules Get Reviewed?
Whenever a lab ships a new agent name, and on a recurring schedule regardless. The agents actually hitting a site can drift from what robots.txt assumes without a single change being made to the file itself.