Here’s a robots.txt setup we see constantly: Claude-User blocked, SearchBot allowed. The logic seems fine. One bot builds the index, the other just grabs a page when asked, so blocking the fetcher should only trim some minor traffic.
Wrong. SearchBot can crawl a page, index it, file it away as something Claude “knows about.” Then someone asks Claude a live question, and Claude never goes back to fetch that page. The page is known. It’s never cited. Those are not the same thing, and treating them as the same thing is the mistake this whole article is about.
Key Takeaways
- SearchBot builds training-data presence over time; Claude-User builds retrieval presence at the moment of a live answer, and the two are governed by separate robots.txt rules.
- A site can be fully open to SearchBot and fully closed to Claude-User at the same time, which lets a brand stay “known” while disappearing from cited answers.
- Firewall and CDN rules can block Claude-User even when robots.txt allows it, so the only reliable check is raw server log data, not the robots.txt file itself.
Two Bots, Two Jobs, One File That Treats Them Differently
ClaudeBot isn’t one thing. It’s two agents wearing the same name, and robots.txt splits them for a reason. SearchBot crawls on its own schedule. No user is waiting on it. It just moves through the web, building Claude’s background sense of who’s authoritative on what. Claude-User works differently. It wakes up only when a real person asks Claude something, and Claude decides it needs to go check a specific page before answering.
Map that onto AEO and the split gets sharper. SearchBot builds what we’d call training-data presence: the slow, cumulative impression a model forms of a brand from broad coverage over months. Claude-User builds retrieval presence: the ability to pull a specific, current page into one specific answer, right now. A brand can max out the first and still score close to zero on the second. Plenty do.
What Actually Breaks When You Block Claude-User
This setup (Claude-User disallowed, SearchBot allowed) is more common than it should be. Usually someone wanted to cut bot traffic and blocked the agent that fires on every live query, assuming the indexer mattered more. That assumption runs backwards. SearchBot already did its job weeks ago. Claude-User is the one standing at the door when the actual question comes in.
Block it, and here’s what happens: Claude still knows the brand exists, pulled from old indexed knowledge. But someone asks about current pricing, or a recent product update, or whether something’s in stock, and memory alone can’t answer that. Claude can’t verify the page live. So it does one of two things. It answers vaguely without naming the brand, or it names a competitor instead, because that competitor’s page is one Claude-User can actually reach. That citation sticks. And it tends to multiply.
Being indexed tells Claude a brand exists. Being fetched is what lets Claude put that brand in front of a real answer.
“We’re Indexed” Is Not The Same Claim As “We Get Cited”
Teams check robots.txt, see no disallow rule for ClaudeBot generally, and move on. Wrong move. SearchBot and Claude-User get evaluated on separate rules, separate requests, separate outcomes. A site can welcome the indexer with open arms and slam the door on the fetcher in the same file. Most quick audits only confirm the first part.
And citations compound. Once Claude cites a source for one query, that source gets more likely to show up again for related ones. It’s a loop. A brand Claude-User can’t reach never gets into that loop in the first place. Meanwhile a competitor with clean fetcher access picks up the first citation, then another, then starts looking like the default answer for the whole category.
total AI citations earned by Moburst’s own brand pages, at an average cited position of 1.77.See the case study
The Gap Widens Quietly, Then It’s A Pattern
Week one, nothing looks wrong. SearchBot keeps crawling, the brand still shows up in general conversation, Share of Voice looks untouched. Give it a month. Share of Citation, how often Claude actually names the brand as a source in a live answer, starts drifting away from Share of Voice. The brand holds its reputation. It loses the actual answers. Those are two different metrics moving in two different directions, and most dashboards only track one of them.
“We’re indexed” sounds like an answer to “can Claude recommend us?” It isn’t. Indexed and fetchable are separate claims, and only the second one turns into a citation inside an actual response.
Stop Guessing. Check The Logs.
Robots.txt tells you what you intended. Server logs tell you what happened. If you want to know whether Claude-User is actually reaching a site, there’s one way to find out: pull the raw access logs, search for its user agent string, and check the response codes. 200s mean it’s getting through. 403s, redirects, or silent timeouts from a firewall rule nobody remembers writing mean it isn’t, no matter what robots.txt says.
- Pull thirty days of logs and filter Claude-User and SearchBot separately. Don’t assume one implies the other. They don’t.
- Check the CDN, the WAF, the rate limiter. Robots.txt compliance and firewall enforcement live in different systems, and one can override the other without telling you.
- Test the pages that actually need to get cited: pricing, specs, comparison pages. Confirm they load clean and fast without client-side JavaScript doing the heavy lifting, since fetchers generally don’t execute scripts the way a browser does.
Fix the access problem first. Structured data, schema, content architecture, none of it matters until the fetcher can actually get to the page.
FAQs
What Is The Difference Between ClaudeBot’s SearchBot And Claude-User?
SearchBot is Anthropic’s indexing crawler. It moves through the web on its own schedule, building Claude’s general sense of a topic. Claude-User only activates when a live question requires Claude to go fetch a specific page in real time, right before it answers.
Why Would Blocking Claude-User Still Matter If SearchBot Is Allowed?
Because they’re governed by separate rules and do separate jobs. A brand can stay fully known to Claude through SearchBot’s crawl while vanishing from every live answer that depends on Claude-User’s fetch. The site looks open. It isn’t, not where it counts.
How Can I Tell If Claude-User Is Being Blocked On My Site?
Pull raw server logs, search for the Claude-User agent string, and check for clean 200 responses. Then check your CDN, WAF, and rate limiter too. Any of those can block the fetcher even when robots.txt says it’s allowed.
Does Blocking Claude-User Affect Share Of Voice?
No. Share of Voice tracks mentions across the web generally and doesn’t move just because one fetcher got blocked. What drops is Share of Citation, how often Claude actually names the brand as a source in a live answer.
