A SaaS company we talked to had spent two weeks on an llms.txt file before asking whether anything actually reads it. That is a fair question, and the honest answer is unsettling: no major AI vendor, not OpenAI, not Google, not Anthropic, has confirmed that its crawlers parse an llms.txt file at all. The file has become a fixture of AEO checklists anyway.
Key Takeaways
- No major AI vendor, including OpenAI, Google or Anthropic, has confirmed that its crawlers read or act on an llms.txt file.
- The file was proposed as a convention, not adopted as a standard, and its actual effect on model behavior is unverified.
- Structured data, clean HTML and citeable content have documented mechanisms behind them. llms.txt currently does not.
What llms.txt Was Supposed to Do
The idea, proposed publicly in 2024, was simple enough. Websites are noisy: navigation, ads, scripts, boilerplate. A large language model trying to understand a site has to wade through all of it. llms.txt was pitched as a clean, curated summary sitting at the root of a domain, a plain-language map of what the site contains and which pages matter most, written for a model rather than a browser.
The analogy people reach for is robots.txt or sitemap.xml. Both of those work because search engines built crawlers that explicitly look for them and act on what they find. That is the part of the analogy that does not hold. robots.txt and sitemap.xml are effective because Google, Bing and every other crawler agreed to honor them and published documentation saying so. llms.txt has the file format. It does not have the agreement.
Why “No Vendor Has Confirmed It” Matters More Than It Sounds
This is not a minor caveat. It is the whole question. A file only functions as a signal if something on the other end is built to read it. Right now, there is no public statement from OpenAI, Google, Anthropic, Perplexity or xAI confirming that their crawlers fetch llms.txt, parse it, or weight anything it contains. Some site owners report crawler hits on the file in server logs, which tells you a bot requested it. It does not tell you the content changed a single output.
That distinction matters because AEO already has two well-understood layers: training-data presence, earned slowly through wide coverage across independent sources, and retrieval presence, earned through structured, current content that a model can fetch at answer time. llms.txt was pitched as a shortcut into retrieval presence. Without a confirmed reader, it is not clear it touches either layer.
growth in monthly AI mentions for Moburst’s own brand, built through structured content, citation building and entity optimization rather than a single file.See the case study
A file only functions as a signal if something on the other end is built to read it.
The Case for Building One Anyway
None of this makes llms.txt useless, and it is worth separating the weak arguments from the reasonable ones.
The weak argument is that it will directly boost citations. There is no evidence for that claim, and repeating it does not create the evidence.
The reasonable argument is smaller and more honest. Writing an llms.txt file forces a team to answer a question worth answering regardless: if a model could only read one page to understand what your company does, what would it say? That exercise produces a clear, jargon-free summary of the business, its products and its positioning. That summary is useful on its own terms, independent of whether any crawler ever fetches the file. Some teams also point to a second, quieter benefit: as a low-cost bet, if a vendor does start honoring the file later, early adopters already have one in place. The cost of writing it is low. The cost of treating it as a strategy is the opportunity cost of the time that could have gone elsewhere.
Where the Evidence Actually Points
If the goal is to influence what ChatGPT, Gemini, Claude, Perplexity or Grok say about a brand, the documented paths run through mechanisms these companies have actually described publicly.
- ChatGPT pulls from across the open web, which means wide, consistent coverage across independent sites shapes what it retrieves and remembers.
- Gemini cross-references Search, YouTube and Scholar, so a page that ranks poorly in Search rarely surfaces in Gemini’s answers either.
- Claude leans on high-authority publications and documentation, which rewards structured, well-sourced content over marketing copy.
- Perplexity cites its sources inline, and its retrieval behavior favors pages with clear structure and current information, not a root-level summary file.
- Grok reads the real-time social web, where recency and discussion volume matter more than any static file at the domain root.
Structured data markup, clean entity signals and content that independent sources reference and agree on all have a documented relationship to how these systems retrieve and cite information. That is the difference between Share of Voice, how often a brand gets mentioned across the web, and Share of Citation, how often a model actually names it as a source. llms.txt has no established link to either. Structured data and citation building do.
Where To Put the Effort Instead
The trust a model places in a brand behaves like a graph of agreement across independent sources, not a single score to be optimized with one file. Citations compound: once a model cites a brand once, the next citation becomes more likely, because the model has effectively treated that source as reliable. Building toward that compounding effect means working on the things with a confirmed mechanism.
That means structured data that correctly describes products, organizations and authorship. It means digital PR that earns mentions on sites a model already trusts. It means content architecture that makes a page’s core claim extractable in a sentence or two, because that is the form models quote. None of this requires abandoning an llms.txt file if one already exists. It does mean not mistaking it for the work.
What To Do Next
If an llms.txt file already exists on a site, leave it. It costs nothing to maintain and there is a small chance it matters later. Do not spend another cycle refining it, and do not report on it as a metric, because there is nothing on the other end confirmed to be reading it.
Redirect that time toward an audit of what AI models currently say about the brand and why. Answerburst’s AI Visibility Audit checks structured data, entity clarity and citation gaps against the mechanisms each model has actually confirmed it uses. That is where the evidence is, and where the next quarter of AEO work should go.
FAQs
Has Any AI Company Confirmed It Reads llms.txt?
No. As of now, OpenAI, Google, Anthropic, Perplexity and xAI have not published documentation confirming their crawlers fetch, parse or act on an llms.txt file, unlike their public statements about sitemap and structured data handling.
Does Having an llms.txt File Hurt a Site?
No. It is a static text file at the root of a domain. It does not conflict with robots.txt, sitemaps or structured data, and maintaining one costs almost nothing beyond the time to write it.
Should a Brand Prioritize llms.txt Over Structured Data?
No. Structured data has a documented relationship to how models like Gemini and Claude retrieve and verify information. llms.txt does not currently have that evidence, so structured data, entity optimization and citation building should come first.
Is llms.txt the Same Idea as Robots.txt?
They share a file-at-the-root format, but robots.txt works because search engines built crawlers that explicitly honor it and documented that behavior. llms.txt has the format without that confirmed vendor commitment.
