A site we reviewed recently had GPTBot fully allowed in robots.txt, a clean llms.txt file listing every product page, and zero citations across any model in six months. The files were technically correct. The content behind them was not structured to be quoted. That gap, between letting crawlers in and giving them something worth citing, is where most AI visibility work actually happens.
Key Takeaways
- robots.txt controls access, llms.txt is a proposed guide to what matters, and neither one makes your content citable on its own.
- Each AI crawler behaves differently: GPTBot, ClaudeBot, PerplexityBot, and Google-Extended each need to be addressed by name, not lumped under a wildcard.
- Blocking AI crawlers to protect your content from training does not stop retrieval-based citation, and the two are frequently confused.
What robots.txt Actually Controls
robots.txt is a request, not a lock. It tells compliant crawlers which paths they may fetch. Every major AI crawler publishes a user-agent string, and you can allow or disallow each one individually in the same file you already use for Googlebot.
The crawlers worth naming directly: GPTBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (which governs Gemini’s use of your content separately from regular Search indexing), and Bytespider (TikTok). A blanket Disallow: / under User-agent: * will catch all of them, which is often not what a site intends. If you want Google to keep indexing you for Search while opting your content out of Gemini’s training set, you need a Google-Extended line specifically, not a general block.
This is also where the training-data-versus-retrieval distinction matters. Blocking GPTBot stops OpenAI from crawling your pages for model training. It does not stop ChatGPT from citing you if a user’s query triggers a live web search and your page ranks in Bing’s index. Those are two separate pipelines with two separate crawlers, and treating them as one setting is the most common robots.txt mistake we see.
What llms.txt Is Actually For
llms.txt is a proposed convention, not a web standard with universal adoption. It lives at your site root, written in Markdown, and lists the pages you consider most important, with a short description of each. The idea is to give a model a curated table of contents instead of forcing it to infer structure from your sitemap or navigation.
No major model has confirmed it systematically ingests llms.txt files at query time the way Googlebot reads sitemap.xml. Adoption is uneven and the specification itself is still evolving. That does not make it worthless. A well-built llms.txt forces you to do something useful regardless of whether any crawler reads the file directly: it makes you state, in plain language, what your site is actually about and which pages answer which questions.
robots.txt decides who gets in. llms.txt is your attempt to tell them what matters once they do. Neither replaces the content doing the actual work of being citable.
Do the Two Files Work Together, or Against Each Other?
They only conflict when they disagree about the same path. If robots.txt disallows a section that llms.txt links to as a priority page, you have told a crawler two things at once: read this, and do not read this. Compliant crawlers will follow robots.txt and skip the page, making your llms.txt entry decorative.
The fix is straightforward but easy to skip: audit both files against each other whenever either changes. Every URL listed in llms.txt should be explicitly allowed for the crawlers you want citing you. If you are blocking ClaudeBot site-wide for licensing reasons but still want Claude to cite your documentation pages, you need a specific allow rule for that path, not a hope that the exception will be understood.
Structuring Content So Crawlers Can Actually Use It
Access without structure gets you a crawl log entry and nothing else. Each engine reads differently, and that should shape how you write, not just where you allow access. Claude leans on high-authority publications and documentation, which rewards clear technical writing with stable URLs. Perplexity cites its sources inline, which rewards content that answers a specific question in a self-contained paragraph a model can quote directly. Gemini cross-references Search, YouTube, and Scholar, so a page that only exists as text is invisible to part of that cross-referencing.
Practical structure choices that help across engines:
- Answer the question in the first two sentences of a section, then explain. Models extract the direct answer more reliably than the supporting argument.
- Use schema markup (FAQPage, Article, Organization) so structured data confirms what your prose already states, rather than leaving a model to infer entity relationships from unstructured text.
- Keep canonical URLs stable. A page that moves or redirects loses accumulated citation history, since a model has no way to know the old and new URLs represent the same source.
Where Teams Get This Wrong
The most common failure is not a missing file. It is treating robots.txt configuration as the finish line instead of the entry requirement. A site can be perfectly open to every AI crawler and still generate zero citations because nothing on it is structured as a clean, quotable answer.
The second common failure is asymmetric blocking done without checking the consequence. A legal or brand team decides to block AI crawlers broadly to prevent training on proprietary content, without distinguishing that from retrieval-based citation. The result is a brand that wanted more visibility in AI answers and configured itself out of eligibility instead.
growth in monthly AI mentions for Moburst’s own brand, with 309 unique pages cited across AI models.See the case study
What to Do Next
Pull your current robots.txt and list every AI-specific user-agent by name: GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Bytespider. Decide deliberately for each one, rather than relying on a wildcard rule you wrote for a different purpose years ago.
Build or update llms.txt with your actual priority pages, ten to twenty entries, each with a one-line description that states what the page answers. Then cross-check every URL in that file against robots.txt to confirm nothing is blocked. Finally, pick your three highest-value pages and rewrite the opening two sentences of each section so they stand alone as a direct answer. That is the part no file at the root of your domain can do for you.
FAQs
Does Blocking GPTBot Stop ChatGPT From Citing My Site?
Not necessarily. Blocking GPTBot stops OpenAI from crawling your pages to train its models. ChatGPT can still surface and cite your content through live web search if your pages are indexed by the search engine ChatGPT queries at answer time. Training access and retrieval access are separate permissions.
Is llms.txt a Confirmed Standard That Models Read?
No. It is a proposed convention with uneven adoption, and no major model has confirmed it reads llms.txt systematically at query time. It is still worth building because the exercise of writing it clarifies your own site structure and priority content.
Should I Block All AI Crawlers if I Do Not Want My Content Used for Training?
You can, but check what you are giving up. A blanket block often removes you from retrieval-based citation as well, even though that is governed by a different mechanism. Address each crawler by name and decide separately whether you care about training use versus citation eligibility.
How Often Should I Update robots.txt and llms.txt?
Review both whenever you restructure URLs, launch new priority content, or a new AI crawler becomes relevant to your traffic. A quarterly check is reasonable for most mid-market sites; anything launching frequent new content should check monthly.
