Ahrefs analyzed 137,000 sites and found that 97% of llms.txt files received zero traffic from AI bots in May 2026. In the same period, adoption of the file grew 8.8x year over year, from 4,088 sites in June 2025 to 36,120 by May 2026 according to Originality.ai's tracking study of three million websites. The picture these two numbers paint is unusual: a standard that almost no AI engine currently reads, being adopted at speed by marketers who believe it matters. This guide cuts through that contradiction.
What llms.txt actually is
llms.txt is a plain Markdown file placed at the root of a website — at yourdomain.com/llms.txt — that gives AI systems a curated map of the site's most important pages, written in a format language models can parse without noise from navigation menus, JavaScript, or cookie banners.
Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed the standard in September 2024. The problem he was solving is real: modern web pages are noisy. A language model trying to understand your product from your homepage has to parse nav bars, cookie banners, JavaScript-rendered content, and promotional copy before it reaches anything useful. An llms.txt file cuts straight to the point. It is a curated briefing, not a sitemap.
The format is four elements. An H1 with your site or brand name. A blockquote with one to three sentences describing what the site covers and who it serves. H2 sections grouping related pages by topic. And annotated links in the format [Title](URL): short description of the page. That is the entire spec. Implementation takes under 30 minutes for most sites.
Does it actually improve AI citations?
No, not today. The evidence on this point is unusually clear for a GEO topic. Ahrefs found that 97% of llms.txt files got zero AI bot traffic in May 2026. OtterlyAI ran a 90-day experiment on their own site and recorded only 84 AI bot visits to /llms.txt out of 62,100 total AI bot hits, which is 0.1% of all AI crawler requests. SE Ranking built an XGBoost model on AI citation data and found that removing the llms.txt variable actually improved prediction accuracy. The file added noise rather than signal. OtterlyAI subsequently removed llms.txt from their GEO audit checklist on the grounds that it was drawing attention away from factors that genuinely move citation frequency.
Only 7.4% of Fortune 500 companies had implemented llms.txt as of March 2026, compared to 92.8% with robots.txt, according to a ProGEO.ai study. None of the major AI platforms — OpenAI, Google, Anthropic, or Meta — have publicly committed to using the file in their production citation systems. The platforms whose search products drive AI citations today focus their published guidance on robots.txt, user agent configuration, and content structure. Not llms.txt.
If you want to improve your AI citation rate today, the highest-leverage moves are allowing AI retrieval crawlers in your robots.txt, adding Organization and Article schema markup, and structuring your content so each section opens with a self-contained 40 to 60 word answer. Those are the signals AI search engines are actually using. llms.txt is not one of them, at least not yet.
Where it does work: the agentic web
There is one context where llms.txt has documented, real-world value: AI coding assistants and developer-facing agents. Tools like Cursor, GitHub Copilot, and Claude retrieve external documentation in real time when developers ask product-specific questions. An llms.txt file helps those tools find the right pages with less token waste. If your site has significant developer traffic or a public API, this case is genuine.
Limy's analysis of over 515 million LLM bot traffic events confirmed that the bots driving AI search citations (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) almost never request /llms.txt. But Profound, a GEO tracking firm, reported that Microsoft and OpenAI crawlers are actively fetching llms.txt and llms-full.txt files. Google included an llms.txt file in its Agents to Agents (A2A) protocol, suggesting internal teams see value in the convention for agent-to-agent communication, even if the Search team does not use it for rankings.
The framing that holds up to scrutiny: llms.txt is not an AI search citation tactic. It is infrastructure for the agentic web, the layer where AI agents act on behalf of users, fetch context, and complete tasks. That layer is growing fast. The cost of shipping the file is trivial. Brands treating it as an optional but clean infrastructure investment are making the right call. Brands treating it as a GEO ranking lever are spending attention they should spend elsewhere.
How to write a correct llms.txt file
The file has four required or strongly recommended elements, in this order. An H1 with your brand or site name — this must be the first element and is the only required element in the spec. A blockquote summary of one to three sentences covering what the site does and who it serves, written factually. H2 sections grouping related pages by topic, ordered by importance rather than your site navigation. And annotated links in the exact format [Title](URL): one-line description per page — the colon-after-link description is the most overlooked element, and agents parse this structure directly so deviations break parsing.
Two common mistakes to avoid. First, do not dump every URL in the file. llms.txt is a curated briefing, not a second sitemap. Pages like legal disclaimers, archive indexes, and tag pages belong in an Optional section at the bottom, if they belong at all. Second, do not set it and forget it. The brand description, product positioning, and pricing details change. Keep the file current with at least a quarterly review.
llms.txt versus robots.txt: the difference that matters
These two files serve completely different purposes, and confusing them costs brands real visibility. robots.txt controls access: it tells crawlers which pages they are allowed to fetch. If AI retrieval crawlers, OAI-SearchBot, PerplexityBot, ClaudeBot, are blocked in your robots.txt, they cannot read your content and you cannot be cited. This is the more consequential file, and Cloudflare's default settings have been blocking these crawlers on new domains since July 2025. Fix robots.txt and Cloudflare before worrying about llms.txt.
llms.txt controls comprehension: it helps AI systems understand what your site covers and where to find specific information, assuming they can already access it. You cannot write a good llms.txt on a site where crawlers are blocked. It is infrastructure that sits on top of access, not a substitute for it.
What to prioritize instead
The factors that drive AI citation rates in 2026, ranked by documented impact: brand mentions across independent third-party sources (Reddit, G2, LinkedIn, industry publications), content structured for extraction with self-contained sections and FAQ schema, technical access with AI crawlers explicitly allowed, and entity clarity via Organization schema with sameAs links to external profiles. GrowthGPT's analysis puts llms.txt as "hygiene, not strategy" — closer to a clean robots.txt than a backlink campaign. If you want the full framework, the 7 GEO levers that actually move citation rates are the right starting point, and a GEO audit will tell you which of those levers is most underdeveloped on your site.
Build the llms.txt file. It takes 30 minutes, costs nothing, and positions you correctly for a future where AI agents do rely on it. But build it after the fundamentals are covered, not instead of them.
Frequently Asked Questions
Does llms.txt improve my rankings in ChatGPT or Perplexity?
No, not based on current data. Ahrefs analyzed 137,000 sites and found 97% of llms.txt files received zero AI bot traffic in May 2026. SE Ranking's machine learning study found that llms.txt presence did not correlate with higher AI citation frequency. No major AI search platform has publicly confirmed they use the file as a citation or ranking signal.
Should I still build one?
Yes, if the cost is low, and for most sites it is. Implementation takes under 30 minutes. The file positions you correctly for the agentic web, where AI coding assistants like Cursor and GitHub Copilot actively read llms.txt files to navigate documentation. Microsoft and OpenAI crawlers have also been observed fetching the file. Ship it after your GEO fundamentals are in order, not instead of them.
How is llms.txt different from robots.txt?
robots.txt controls which pages AI crawlers are allowed to access. If OAI-SearchBot or PerplexityBot are disallowed in your robots.txt, they cannot read your content regardless of what your llms.txt says. llms.txt controls comprehension, helping AI systems understand what your site covers once access is granted. Fix crawler access first. Then build llms.txt on top of it.
What is the correct format for a llms.txt file?
The file uses Markdown. It starts with an H1 containing your site or brand name. Below that, a blockquote with a one to three sentence factual description of what the site does. Then H2 sections grouping related pages, each with links in the format: [Title](URL): one-line description. Agents parse this format directly — deviations break parsing. Keep it curated, not exhaustive.
How is llms-full.txt different from llms.txt?
llms.txt is the index: a curated map of your most important pages with short descriptions, designed for fast navigation. llms-full.txt is the library: your full site content compiled into a single Markdown document for deep AI ingestion with more detail per page. Most sites only need llms.txt. Documentation-heavy SaaS products and developer platforms with large API references benefit from maintaining both and keeping them synchronized.
Know what AI engines actually say about your brand
llms.txt tells AI systems about your site. Knowing what ChatGPT, Perplexity, and Gemini actually say about your brand is a different problem. VizibleAI monitors that daily across six AI engines, tracking your mention rate, citation position, sentiment, and competitive share of voice. Start your free 7-day trial — no credit card required.