llms.txt is a plain text file at the root of a website that lists the site's most important pages for AI tools. Each link can carry a short note on what the page holds. The file is optional and quick to write.
Almost nothing asks for it. In an Ahrefs study of server logs, 97% of llms.txt files got no requests at all in May 2026. Google says its search ignores the file. So write it last. First check the two things that decide whether AI bots can reach your site at all. Those are your robots.txt file and the bot settings of your CDN, the service that delivers your pages, such as Cloudflare. Then fix your schema and page structure.
We are an agency that gets brands recommended by AI assistants, and on client sites we work in that order. We keep an llms.txt file on our own site as well, because it takes minutes and audit tools look for it.
What is llms.txt, in plain words?
llms.txt is a list of links to your best pages, and it always sits at the same address: your domain followed by /llms.txt, as in example.com/llms.txt. The "llms" in the name stands for large language models, the AI behind ChatGPT, Claude and Gemini. A bot reads the file only if it asks for that address on purpose, the same way it asks for any page.
The file is most useful on sites that AI agents work with: developer documentation and apps with an API. Peec AI, which tracks how brands show up in AI answers, says the file "seems like a good bet" for an app or an API that AI agents use, and calls it "a distraction without any upside" for anyone chasing traffic from ChatGPT and Perplexity.
What does the llms.txt spec say?
Just one line is required: a top heading with the name of your site. Everything else is optional in the proposal that defines llms.txt. Jeremy Howard, a co-founder of fast.ai, published it at llmstxt.org in September 2024.
The proposal sets no fixed length. It says the file "stays small enough to fit in context", meaning an AI model can read all of it in one go. The detail stays on the linked pages. Its author expected llms.txt to help most while an AI tool is answering a question, and less when a model is being trained.
Is llms.txt actually used?
Barely. Ahrefs studied the server logs of 137,210 websites that use its analytics. A little more than one in four of them had a valid llms.txt file. In May 2026, 97% of those files got no requests at all.
Source: Ahrefs, server logs of 137,210 websites, May 2026.
Audit tools and each kind of AI bot sent these shares of all llms.txt requests in May 2026:
Who asked for the file | What they do | Share of all llms.txt requests, May 2026 |
|---|---|---|
SEO audit tools | Crawl sites for SEO health checks, with no special interest in llms.txt | 21.7% |
AI agents and the tools built for them, such as Claude Code | Act on a person's behalf, for example while writing code | 10.5% |
AI-readiness scoring tools | Scan sites and score how ready they are for AI search | 5.8% |
AI training crawlers, such as GPTBot and ClaudeBot | Collect pages to train future AI models | 5.3% |
User-triggered fetchers, such as ChatGPT-User and Claude-User | Open a page because one person asked one question | 2.5% |
AI search bots, such as OAI-SearchBot and PerplexityBot | Fetch pages to answer live questions in AI search | 1.1% |
Tools that grade websites asked for llms.txt far more often than the AI search bots that fetch pages for the answers your buyers read. Ahrefs also counted unidentified bots, general web crawlers and technology profilers, and each of those asked for the file more often than AI agents did.
"Zero requests came from AI bots for llms.txt files that don't exist," the Ahrefs study adds. On a site without the file, "they never go looking."
Is llms.txt mandatory, and does it help SEO?
No on both counts. llms.txt is optional, and it serves no purpose in Google SEO: Google says the file "will neither harm nor help" your visibility or rankings in Google Search. Google's guide to its generative AI search features, updated in July 2026, tells site owners they "don't need to create new machine readable files, AI text files, markup, or Markdown" to appear there, "as Google Search itself doesn't use them".
Google's John Mueller went further on Reddit, as quoted by Ahrefs. He wrote that "none of the AI services have said they're using LLMs.TXT", and that "you can tell when you look at your server logs that they don't even check for it". He compared the file to the keywords meta tag, calling it "what a site-owner claims their site is about".
A second part of the llms.txt proposal asks for a Markdown copy of each page that holds information AI agents might need, at the same address with .md added. Peec AI warns that the copies "might actually hurt by creating duplicate content".
Chrome's Lighthouse audit does look for the file. When a site has none, Lighthouse marks the check as not applicable, because "providing the file is optional at the moment".
What should you fix before writing llms.txt?
Start with anything that keeps AI bots from reaching and reading your pages. Letting the bots in is the first step of generative engine optimization (GEO), the work of getting a brand named and cited in AI answers.
Open robots.txt to the AI bots. Check for Disallow lines under OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot and Google-Extended. The search bots have to stay open, since OpenAI states that sites which opt out of OAI-SearchBot "will not be shown in ChatGPT search answers". Training bots, GPTBot and ClaudeBot among them, are a business choice, and some companies block them on purpose to keep their content out of AI training. We advise keeping them open: a model that cannot read your site learns about you from other sites. Our list of AI crawlers gives each bot's job.
Check the blocks robots.txt cannot see. Your CDN and your server's firewall can both turn a bot away before it reads robots.txt. Read the security log in your Cloudflare or hosting dashboard. A test from your own computer that borrows GPTBot's name only catches a rule on that name. It does not come from the addresses OpenAI publishes for its bots, so a block on the real bot shows up only in that log.
Add schema that matches the page. Google says schema is still "a good idea", because it helps pages qualify for rich results in Google Search. We rank schema above llms.txt for a second reason: schema lives inside the pages bots already read, while llms.txt is a separate file they rarely ask for.
Put the answer where a bot can read it. Vercel found in December 2024 that OpenAI's and Anthropic's crawlers did not run JavaScript, so text that appeared only after scripts ran was invisible to them. Answer each page's main question near the top, in text that is already there when the page loads. Our guide to ranking on ChatGPT shows how these pages are built.
Write llms.txt last. It is a short job once bots can get in and read your pages.
The order comes from what we find on client sites. One client's site used Cloudflare, and Cloudflare's setting to block AI training bots was switched on for every page. The client published an llms.txt file, an open robots.txt and new schema, and OpenAI's bots still could not get in.
The client's Cloudflare security log then showed dozens of requests from OpenAI's bots refused in a single day. Some of them were triggered by people asking ChatGPT a question, so the setting was catching more than training crawls. Two weeks after the llms.txt file went live, the client switched the setting off in the Cloudflare dashboard, and OpenAI's bots could get in.
robots.txt can also change without anyone noticing. A routine site update can put AI bots on Disallow while every page looks the same, so read the file again after each release.
How do you write an llms.txt file in ten minutes?
Open a plain text editor and gather the few pages that matter most. Those are the pages that say who you are, what you sell, what it costs, where your proof is and how to reach you.
Put your company name on the first line, after one hash sign (#).
Add one line that starts with a greater-than sign (>) and says what you do and for whom.
Add a few bullet points of plain facts, such as what you sell, where you sell it and how to reach you. Each fact must match what your site says.
Add sections under headings that start with two hash signs (##), such as Product and Proof. Each section lists links to pages, with one line on what each page covers.
End with a section called Optional, for pages that are useful but secondary, such as the blog.
Save the file as llms.txt at the root of your site. Open your domain followed by /llms.txt in a browser and check that it loads, since Lighthouse flags a page when the file returns a server error.
A short example for a made-up company:
Our own file, at agenzy.lt/llms.txt, follows the same outline, with sections for services, proof, guides and contact.
Questions people ask about llms.txt
Does ChatGPT read llms.txt?
OpenAI's bots rarely ask for the file. In the Ahrefs study of May 2026, only about 1,100 sites saw any requests for their llms.txt file. GPTBot, the OpenAI bot that collects pages for training, made 4.51% of those requests. OAI-SearchBot, the bot behind ChatGPT search, made 0.74%. When ChatGPT searches the web, it fetches and reads your pages themselves.
How do I generate an llms-full.txt file?
Documentation platforms like Mintlify and GitBook build it automatically. llms-full.txt puts the full text of a documentation site into one file, so a site without developer docs rarely needs one. Mintlify says it developed the format with Anthropic and that it was later included in the official llms.txt proposal, though llmstxt.org does not mention it.
What is the difference between llms.txt and robots.txt?
Only one of them sets rules. robots.txt tells each bot which parts of a site it may visit. OpenAI uses it as the opt-out for its search and training bots, so a single Disallow line can keep one of those bots off your whole site. llms.txt is a reading list of your main pages, which a bot may open or ignore.
How often should I update llms.txt?
Update llms.txt whenever a price changes or a key page moves, such as your pricing page or a service page. Few tools read the file, but SEO audit tools and the occasional AI agent do, and they pick up whatever it says, even a price you changed on your site months ago.
Is it worth paying someone to create an llms.txt file?
Save your money: anyone who knows the site can write one by hand, and there is no evidence that the file raises AI visibility. Put it toward opening your site to AI bots, writing pages worth citing and tracking which ones AI answers cite. Agenzy's GEO service starts from 5,000 EUR a month, ex VAT, and the full scope is on our GEO service page.
About Agenzy
Agenzy is a GEO agency based in Vilnius, working with brands in the US, the UK and across Europe. We get brands named and recommended inside ChatGPT, Gemini, Google AI Overviews, Perplexity, Claude and Copilot. We are an official Peec AI partner. As of September 2026: 500,000+ AI chats analysed, 15,000+ prompts tracked, 150+ audits completed, 1,000,000+ EUR generated for clients by AI search. Dated cases sit at agenzy.lt/case-studies.
