llms.txt: what it is, who actually reads it, and what to fix first

llms.txt is a small text file that points AI tools to a website's most important pages. Server logs from 137,210 websites show that almost nothing asks for it, and Google says its search ignores it. What the file holds, who reads it, and the four fixes that matter more, with one client site where a Cloudflare setting kept OpenAI's bots out.

llms.txt is a small text file that points AI tools to a website's most important pages. Server logs from 137,210 websites show that almost nothing asks for it, and Google says its search ignores it. What the file holds, who reads it, and the four fixes that matter more, with one client site where a Cloudflare setting kept OpenAI's bots out.

Read Time:

Read Time:

Read Time:

~10 min

~10 min

Written by:

Written by:

Written by:

Reviewed by:

Reviewed by:

Reviewed by:

Reviewed on:

Reviewed on:

Reviewed on:

llms.txt: what it is, who actually reads it, and what to fix first

llms.txt is a plain text file at the root of a website that lists the site's most important pages for AI tools. Each link can carry a short note on what the page holds. The file is optional and quick to write.

Almost nothing asks for it. In an Ahrefs study of server logs, 97% of llms.txt files got no requests at all in May 2026. Google says its search ignores the file. So write it last. First check the two things that decide whether AI bots can reach your site at all. Those are your robots.txt file and the bot settings of your CDN, the service that delivers your pages, such as Cloudflare. Then fix your schema and page structure.

We are an agency that gets brands recommended by AI assistants, and on client sites the technical work goes into those fixes.

What is llms.txt, in plain words?

llms.txt is a list of links to your best pages, and it always sits at the same address: your domain followed by /llms.txt, as in example.com/llms.txt. The "llms" in the name stands for large language models, the AI behind ChatGPT, Claude and Gemini. A bot reads the file only if it asks for that address on purpose, the same way it asks for any page.

The file is most useful on sites that AI agents work with: developer documentation and apps with an API. Peec AI, which tracks how brands show up in AI answers, says the file "seems like a good bet" for an app or an API that AI agents use, and calls it "a distraction without any upside" for anyone chasing traffic from ChatGPT and Perplexity.

What does the llms.txt spec say?

Just one line is required: a top heading with the name of your site. Everything else is optional in the proposal that defines llms.txt. Jeremy Howard, a co-founder of fast.ai, published it at llmstxt.org in September 2024.

The proposal sets no fixed length. It says the file "stays small enough to fit in context", meaning an AI model can read all of it in one go. The detail stays on the linked pages. Its author expected llms.txt to help most while an AI tool is answering a question, and less when a model is being trained.

Is llms.txt actually used?

Barely. Ahrefs studied the server logs of 137,210 websites that use its analytics. A little more than one in four of them had a valid llms.txt file. In May 2026, 97% of those files got no requests at all.

97% of llms.txt files got no requests in May 2026, and AI search bots sent 1.1% of the requests that came

Source: Ahrefs, server logs of 137,210 websites, May 2026.

Audit tools and each kind of AI bot sent these shares of all llms.txt requests in May 2026:

Who asked for the file

What they do

Share of all llms.txt requests, May 2026

SEO audit tools

Crawl sites for SEO health checks, with no special interest in llms.txt

21.7%

AI agents and the tools built for them, such as Claude Code

Act on a person's behalf, for example while writing code

10.5%

AI-readiness scoring tools

Scan sites and score how ready they are for AI search

5.8%

AI training crawlers, such as GPTBot and ClaudeBot

Collect pages to train future AI models

5.3%

User-triggered fetchers, such as ChatGPT-User and Claude-User

Open a page because one person asked one question

2.5%

AI search bots, such as OAI-SearchBot and PerplexityBot

Fetch pages to answer live questions in AI search

1.1%

Tools that grade websites asked for llms.txt far more often than the AI search bots that fetch pages for the answers your buyers read. Ahrefs also counted unidentified bots, general web crawlers and technology profilers, and each of those asked for the file more often than AI agents did.

"Zero requests came from AI bots for llms.txt files that don't exist," the Ahrefs study adds. On a site without the file, "they never go looking."

Is llms.txt mandatory, and does it help SEO?

No on both counts. llms.txt is optional, and it serves no purpose in Google SEO: Google says the file "will neither harm nor help" your visibility or rankings in Google Search. Google's guide to its generative AI search features, updated in July 2026, tells site owners they "don't need to create new machine readable files, AI text files, markup, or Markdown" to appear there, "as Google Search itself doesn't use them".

Google's John Mueller went further on Reddit, as quoted by Ahrefs. He wrote that "none of the AI services have said they're using LLMs.TXT", and that "you can tell when you look at your server logs that they don't even check for it". He compared the file to the keywords meta tag, calling it "what a site-owner claims their site is about".

A second part of the llms.txt proposal asks for a Markdown copy of each page that holds information AI agents might need, at the same address with .md added. Peec AI warns that the copies "might actually hurt by creating duplicate content".

Chrome's Lighthouse audit does look for the file. When a site has none, Lighthouse marks the check as not applicable, because "providing the file is optional at the moment".

What should you fix before writing llms.txt?

Start with anything that keeps AI bots from reaching and reading your pages. Letting the bots in is the first step of generative engine optimization (GEO), the work of getting a brand named and cited in AI answers.

  1. Open robots.txt to the AI bots. Check for Disallow lines under OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot and Google-Extended. The search bots have to stay open, since OpenAI states that sites which opt out of OAI-SearchBot "will not be shown in ChatGPT search answers". Training bots, GPTBot and ClaudeBot among them, are a business choice, and some companies block them on purpose to keep their content out of AI training. We advise keeping them open: a model that cannot read your site learns about you from other sites. Our list of AI crawlers gives each bot's job.

  2. Check the blocks robots.txt cannot see. Your CDN and your server's firewall can both turn a bot away before it reads robots.txt. Read the security log in your Cloudflare or hosting dashboard. A test from your own computer that borrows GPTBot's name only catches a rule on that name. It does not come from the addresses OpenAI publishes for its bots, so a block on the real bot shows up only in that log.

  3. Add schema that matches the page. Google says schema is still "a good idea", because it helps pages qualify for rich results in Google Search. We rank schema above llms.txt for a second reason: schema lives inside the pages bots already read, while llms.txt is a separate file they rarely ask for.

  4. Put the answer where a bot can read it. Vercel found in December 2024 that OpenAI's and Anthropic's crawlers did not run JavaScript, so text that appeared only after scripts ran was invisible to them. Answer each page's main question near the top, in text that is already there when the page loads. Our guide to ranking on ChatGPT shows how these pages are built.

  5. Write llms.txt last. It is a short job once bots can get in and read your pages.

Ladder of five technical checks, with robots.txt access first and llms.txt last

The order comes from what we find on client sites. One client's site used Cloudflare, and Cloudflare's setting to block AI training bots was switched on for every page. The client published an llms.txt file, an open robots.txt and new schema, and OpenAI's bots still could not get in.

The client's Cloudflare security log then showed dozens of requests from OpenAI's bots refused in a single day. Some of them were triggered by people asking ChatGPT a question, so the setting was catching more than training crawls. Two weeks after the llms.txt file went live, the client switched the setting off in the Cloudflare dashboard, and OpenAI's bots could get in.

robots.txt can also change without anyone noticing. A routine site update can put AI bots on Disallow while every page looks the same, so read the file again after each release.

How do you write an llms.txt file in ten minutes?

Open a plain text editor and gather the few pages that matter most. Those are the pages that say who you are, what you sell, what it costs, where your proof is and how to reach you.

  1. Put your company name on the first line, after one hash sign (#).

  2. Add one line that starts with a greater-than sign (>) and says what you do and for whom.

  3. Add a few bullet points of plain facts, such as what you sell, where you sell it and how to reach you. Each fact must match what your site says.

  4. Add sections under headings that start with two hash signs (##), such as Product and Proof. Each section lists links to pages, with one line on what each page covers.

  5. End with a section called Optional, for pages that are useful but secondary, such as the blog.

  6. Save the file as llms.txt at the root of your site. Open your domain followed by /llms.txt in a browser and check that it loads, since Lighthouse flags a page when the file returns a server error.

A short example for a made-up company:

# Example Co

> Example Co makes appointment software for independent dental clinics in the US.

- Customers: independent dental clinics
- Support: help@example.com

## Product
- [Features](https://example.com/features): What the software does, screen by screen
- [Pricing](https://example.com/pricing): Plans, prices and what each one includes

## Proof
- [Customer stories](https://example.com/customers): Named clinics and their results

## Optional
- [Blog](https://example.com/blog)

# Example Co

> Example Co makes appointment software for independent dental clinics in the US.

- Customers: independent dental clinics
- Support: help@example.com

## Product
- [Features](https://example.com/features): What the software does, screen by screen
- [Pricing](https://example.com/pricing): Plans, prices and what each one includes

## Proof
- [Customer stories](https://example.com/customers): Named clinics and their results

## Optional
- [Blog](https://example.com/blog)

# Example Co

> Example Co makes appointment software for independent dental clinics in the US.

- Customers: independent dental clinics
- Support: help@example.com

## Product
- [Features](https://example.com/features): What the software does, screen by screen
- [Pricing](https://example.com/pricing): Plans, prices and what each one includes

## Proof
- [Customer stories](https://example.com/customers): Named clinics and their results

## Optional
- [Blog](https://example.com/blog)

Questions people ask about llms.txt

Does ChatGPT read llms.txt?

OpenAI's bots rarely ask for the file. In the Ahrefs study of May 2026, only about 1,100 sites saw any requests for their llms.txt file. GPTBot, the OpenAI bot that collects pages for training, made 4.51% of those requests. OAI-SearchBot, the bot behind ChatGPT search, made 0.74%. When ChatGPT searches the web, it fetches and reads your pages themselves.

How do I generate an llms-full.txt file?

Documentation platforms like Mintlify and GitBook build it automatically. llms-full.txt puts the full text of a documentation site into one file, so a site without developer docs rarely needs one. Mintlify says it developed the format with Anthropic and that it was later included in the official llms.txt proposal, though llmstxt.org does not mention it.

What is the difference between llms.txt and robots.txt?

Only one of them sets rules. robots.txt tells each bot which parts of a site it may visit. OpenAI uses it as the opt-out for its search and training bots, so a single Disallow line can keep one of those bots off your whole site. llms.txt is a reading list of your main pages, which a bot may open or ignore.

How often should I update llms.txt?

Update llms.txt whenever a price changes or a key page moves, such as your pricing page or a service page. Few tools read the file, but SEO audit tools and the occasional AI agent do, and they pick up whatever it says, even a price you changed on your site months ago.

Is it worth paying someone to create an llms.txt file?

Save your money: anyone who knows the site can write one by hand, and there is no evidence that the file raises AI visibility. Put it toward opening your site to AI bots, writing pages worth citing and tracking which ones AI answers cite. Agenzy's GEO service starts from 5,000 EUR a month, ex VAT, and the full scope is on our GEO service page.

Emilis Zabilius, CEO and co-founder of Agenzy

Written by

CEO & Co-founder, Agenzy

Emilis Zabilius is the CEO and co-founder of Agenzy, a GEO agency that gets brands named in ChatGPT, Gemini, Perplexity and Google AI Overviews answers. He leads sales and marketing and writes the agency’s research and guides on AI search, including the 2026 study of 2,283,923 AI citations. Agenzy is his third company, after an education startup for children and a marketing agency he co-founded with Rokas Alešiūnas.

Reviewed by 

, 

COO & Co-founder, Agenzy

About Agenzy

Agenzy is a GEO and AI SEO agency based in Europe, working with brands across Europe, the UK and the US. We get brands named and recommended inside ChatGPT, Gemini, Google AI Overviews, Perplexity, Claude and Copilot. We are an official Peec AI partner. As of October 2026: 1,000,000+ AI chats analysed, 15,000+ prompts tracked, 200+ audits completed, 1,000,000+ EUR generated for clients by AI search. Dated cases sit at agenzy.lt/case-studies.

EN

Emilis Zabilius

Emilis Zabilius

CEO & Co-Founder, Agenzy

Where does your brand stand in AI search?

Free check of how often ChatGPT, Gemini and Google AI Overviews name you, and whether they can read your site.

Where does your brand stand in AI search?

Free check of how often ChatGPT, Gemini and Google AI Overviews name you, and whether they can read your site.

Our help

Related Services

Related Services

Related Services

Services related to the blog.

Services related to the blog.

Services related to the blog.

Keep up with what matters.

Simple, useful ideas on GEO, content and growth, shared on LinkedIn.

Next step

Check your own AI search potential

1

Free check

We check whether AI can read your site and how often it names you.

2

30-minute intro call

We map your goals and give a first read on the gaps.

3

Deep dive + plan

A 60-minute session on your buyer questions and what we would build first.

Emilis Zabilius

Emilis Zabilius

CEO & Co-Founder

Focus · AI search and ChatGPT ads

Based · Europe

Rokas Alešiūnas

Rokas Alešiūnas

COO & Co-Founder

Focus · Strategy and clients

Based · Europe

Keep reading

More from Agenzy

recommended reads

Guide

What does ChatGPT Business Hub measure?

OpenAI's Business Hub lets a business claim its ChatGPT profile and see 28 days of mentions, clicks and topics. This post compares the Hub with Search Console and sets out what to do this week.

read

Guide

How to get cited by Claude in 2026

How to get cited by Claude in 2026 depends on three things. Claude searches the web only for some prompts, mostly comparisons and current facts, and answers the rest from memory without sources.

read

Guide

How to rank in Perplexity in 2026

This article explains how to rank in Perplexity in 2026, which for a B2B company means being recommended: your site cited and your company named.

read

Guide

Does ChatGPT use Bing or Google in 2026?

ChatGPT pulls results from Microsoft, from Google through scrapers and from Labrador, an index OpenAI built itself. Here is the evidence, checked on 2026-10-17, and what it means for your site.

read

Guide

Which are the best GEO agencies in the US in 2026?

We scored sixteen GEO agencies on eleven things a US buyer can check on their own pages, from prompt tracking and attribution to dated cases and price. Agenzy leads at 21 of 22, followed by Omniscient Digital at 20 and Seer Interactive at 19, with a source behind every cell.

read

Guide

Generative engine optimization services for B2B SaaS: what a GEO agency does in the first 90 days

Month one writes the tracked prompt set, measures the baseline per engine and opens the site to AI crawlers. Month two ships the pages and the off-site placements. Month three delivers a per-engine report against that baseline. This is our calendar, with the client input each row needs, the check that proves it landed, what day 90 is worth in Google demand and attributed leads, and a section on what 90 days does not buy.

read

Guide

What an AI visibility audit of a B2B SaaS company actually finds

One real AI visibility audit of one real B2B SaaS company. We have masked the company and every number is exactly as measured. Seventy-five buyer prompts, 2,153 AI answers across ChatGPT, Perplexity and Google AI Overviews, four markets, one day in July 2026. The company was named in 23.59% of answers and in 1 of 376 answers to problem-shaped questions.

read

Guide

How to choose a GEO agency when every proposal reads the same

Every GEO proposal offers an AI visibility audit, citation-ready content, technical work and digital PR, because that stack is identical across the market. We reviewed 22 agencies' public pages, directory profiles and press coverage in September 2026. This is what to ask instead, what a real answer sounds like, and the one request a relabeled SEO retainer cannot fill.

read

Agenzy logo

Get started

Let’s make AI recommend you.

Book a free 30 min call and we’ll show you where you appear in AI answers today.

3D phone showing a ChatGPT style answer to what is the best CRM for a small business
Agenzy logo

Get started

Let’s make AI recommend you.

Book a free 30 min call and we’ll show you where you appear in AI answers today.

3D phone showing a ChatGPT style answer to what is the best CRM for a small business
Agenzy logo

Get started

Let’s make AI recommend you.

Book a free 30 min call and we’ll show you where you appear in AI answers today.

3D phone showing a ChatGPT style answer to what is the best CRM for a small business