A SaaS company blocked every OpenAI crawler it found listed in its robots.txt, GPTBot included, assuming it was protecting its content from being used to train future models. Six weeks later, its answer share in ChatGPT dropped to zero. The same wildcard rule that blocked GPTBot also blocked OAI-SearchBot and ChatGPT-User, the crawlers that fetch pages live to answer a specific question. A robots.txt policy needs to treat those two jobs as separate rules.
Key Takeaways
- GPTBot trains future versions of the model. OAI-SearchBot and ChatGPT-User retrieve pages at the moment a user asks a question, and blocking them removes a brand from live citations regardless of what happens with training.
- Many robots.txt files block all three crawlers by accident, because a single disallow rule written for “OpenAI” catches every user-agent string that contains it.
- Training exclusion and retrieval participation are different decisions with different consequences, and a policy that does not separate them makes both decisions by default instead of on purpose.
What GPTBot, OAI-SearchBot, and ChatGPT-User Actually Do
OpenAI runs three named crawlers, and they do not serve the same purpose. GPTBot crawls the web to gather material that may be used in training future models. It runs on its own schedule, unconnected to any single user’s question, and its access has nothing to do with whether a page shows up in an answer today.
OAI-SearchBot and ChatGPT-User are retrieval crawlers. They fetch a page because ChatGPT is actively answering a query and needs current content to cite. OAI-SearchBot powers the search feature inside ChatGPT. ChatGPT-User fires when a user’s prompt triggers a live browse, the same mechanism that lets the model pull a price, a date, or a statistic that did not exist when it was last trained. Block either one and the model simply cannot see the page at the moment it needs it.
Why Blocking All Three Kills Citations, Not Just Training
A brand that blocks GPTBot only has opted out of training-data presence. That is a defensible choice, and plenty of publishers make it deliberately. The problem starts when the same disallow rule, often written broadly to save time, also blocks OAI-SearchBot and ChatGPT-User. At that point the brand has opted out of retrieval presence too, and retrieval presence is what determines whether ChatGPT can cite the page in an answer given right now.
This distinction matters because Share of Citation and Share of Voice are not the same metric. A brand can rank on page one of Google, get mentioned constantly across the web, and still register near zero citations inside AI answers if its retrieval crawlers are blocked. Traffic volume and brand mentions do not compensate for a closed door at the exact moment a model tries to fetch the page.
growth in AI brand presence for NewDay USA, tracked weekly across five engines over six months.See the case study
The Retrieval Layer Depends on Being Fetchable in Real Time
AEO has two layers, and they are earned differently. Training-data presence builds slowly, through wide coverage across the web that eventually gets folded into a model’s weights. Retrieval presence works on a much shorter clock: it depends on whether a crawler can reach a page right now, parse it, and pull a fact or a quote into the answer being generated.
Blocking GPTBot protects your content from training. Blocking OAI-SearchBot protects you from ever being the answer.
Citations compound once they start. A page that gets cited by ChatGPT becomes slightly more likely to be cited again on a related query, because the model has already established it as a trusted source for that topic. Blocking the retrieval crawlers does not just cost one missed citation. It cuts off the loop before it can start.
Writing a Robots.txt Policy That Separates the Two Decisions
The fix is not to allow or block “OpenAI” as a single unit. It is to write a rule per user-agent, so each crawler gets a decision that matches its actual function.
A policy that excludes training but keeps retrieval open reads like this:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
A brand that wants no OpenAI presence at all writes the mirror image, disallowing all three explicitly rather than relying on a wildcard that might miss one or catch one it should not. The point of writing them out individually, rather than grouping them under one rule, is that the decision becomes visible and auditable. Anyone reviewing the file later can see exactly which crawler was excluded and why, instead of inheriting a blanket rule nobody remembers writing.
The same logic extends past OpenAI. Gemini, Claude, and Perplexity each run their own retrieval crawlers separate from whatever indexes for training, and a robots.txt file written for one platform’s naming convention will not automatically cover the others. Each needs its own line.
What to Do Next
Pull the current robots.txt file and check it line by line for GPTBot, OAI-SearchBot, and ChatGPT-User as separate entries. If a wildcard rule or a grouped disallow is catching more than one, split it out immediately, even if the intended policy stays the same. The goal is a file where every crawler’s access was a decision, not a side effect.
Then check server logs for actual crawler activity over the past thirty days. A brand can have a permissive robots.txt file and still see zero OAI-SearchBot hits, which points to a different problem: the content itself is not structured for a retrieval crawler to parse quickly, even when it is allowed in.
FAQs
Does Blocking GPTBot Stop a Brand From Being Cited in ChatGPT Answers?
No. GPTBot only affects whether content is gathered for training future model versions. Citations inside live ChatGPT answers depend on OAI-SearchBot and ChatGPT-User, which are separate crawlers with separate robots.txt entries.
Can a Single Robots.txt Rule Accidentally Block All Three OpenAI Crawlers?
Yes. A disallow rule written broadly, or a wildcard pattern intended to catch one user-agent, often matches all three because GPTBot, OAI-SearchBot, and ChatGPT-User all contain overlapping naming conventions. Each crawler needs its own explicit line to avoid this.
Is There a Reason to Block Training but Allow Retrieval?
Yes. Some brands want their published content excluded from what shapes future model weights, for competitive or legal reasons, while still wanting to be citable in answers generated today. Writing separate rules for GPTBot versus OAI-SearchBot and ChatGPT-User makes that split possible.
How Often Should a Robots.txt File Be Reviewed for This Issue?
Whenever a new AI crawler enters wide use, and at minimum once a quarter, since crawler naming conventions and platform behavior change faster than most teams update a static file.
