What Is llms.txt? What It Does, Whether It Helps AI Search, and How to Create One

B2B marketing and SEO teams keep hearing the same advice: publish a Markdown guide for AI systems at the root of your site so ChatGPT, Gemini, Perplexity and Google AI Overviews will cite you. Most of what's written about the file either sells it as a shortcut to AI visibility or dismisses it as pointless.

So what is llms.txt? It's a Markdown file placed at the root of a website that gives large language models a short, curated guide to the site's most useful pages. It's quick to create and unlikely to cause harm. However, as of October 2026, the crawler and search documentation from OpenAI, Perplexity and Google doesn't say any of them uses it to decide which sources to cite.

This guide covers how the file works and what the evidence actually shows. It walks through a six-step build with a worked example, explains how the file differs from robots.txt and sitemap.xml, helps you decide whether your team should bother, and shows how to test the file before it goes live.

What Is llms.txt?

It's a plain Markdown file published at a website's root that gives large language models and AI agents a curated summary of the site, plus links to its most important pages. It was proposed in 2024 as a voluntary convention. It is not an official web standard, and it does not control crawling.

The proposal was published by Jeremy Howard, co-founder of Answer.AI, in September 2024 at llmstxt.org. It addresses a practical problem. Language models work within limited context windows, and a typical HTML page carries navigation, scripts, cookie banners and template boilerplate around the content that matters. A compact Markdown index lets a model or agent reach the right pages quickly at inference time, meaning the moment a user asks a question. It is not built for model training.

Despite how often people call it a standard, it has no W3C or IETF status. It is a community proposal that anyone can adopt or ignore. Compare that with robots.txt, which was formalized as IETF RFC 9309 in 2022 after decades of informal use. Many people also search for "llm.txt", but the filename in the proposal is plural: llms, not llm.

In practice, the file is one small, optional item within answer engine optimization, the work of making your content easy for AI systems to find, understand and quote. It is not a strategy on its own.

The same applies to generative engine optimization (GEO), which focuses on how generative models select and summarize sources. The file can support that work, but the pages it points to do the heavy lifting.

How the File Works: Location, Format and llms-full.txt

The file is deliberately simple: one location, a handful of Markdown conventions and an optional companion file. Only one element is actually required, so most of what follows is about doing the optional parts well.

Where the file goes?

Where to place the file on your website has a simple answer: the root of the domain, directly under https://www.yourdomain.com/, using the plural filename noted above. If you put it somewhere else, such as inside a /blog/ folder, tools that look at the root won't find it.

Enterprise sites need one extra decision. Each subdomain (docs., support., investors.) is treated as a separate site, so each needs its own file if you want it covered. The proposal also allows files at sub-paths, which suits documentation sets that live in a folder. Serve the file as plain text or Markdown (text/plain or text/markdown) in UTF-8.

Check Our Answer Engine Optimization Services

The format, element by element

The file's format uses standard Markdown in a fixed order:

• H1 with the site or project name. This is the only required element.

• A blockquote (>) with a short summary of what the organization does and for whom.

• Optional free-text paragraphs for context, such as what the site covers and what it doesn't.

• H2 sections, each holding a Markdown list of links in the form - [Page name](URL): one-line description.

• An H2 named "Optional" for secondary links that tools can skip when context is tight.

The proposal also suggests that pages offer clean Markdown versions at the same URL with .md appended. In practice, most B2B content management systems don't do this by default, and linking to normal HTML URLs is common practice.

What llms-full.txt is

llms-full.txt is a separate, optional file that contains the full text of your key pages in one Markdown document, instead of links to them. It isn't defined in the original llmstxt.org proposal. It's a widely used convention, popularized largely by documentation platforms.

It earns its place on developer documentation and API references, where an agent benefits from having the whole reference in one fetch. On a marketing site it usually backfires. It becomes a huge duplicate of your site that goes out of date the first time someone edits a service page.

Does the File Help AI Search or SEO? What the Evidence Shows?

This is the question that matters most, and the honest answer depends on who is reading the file. Three audiences matter here: Google Search (including AI Overviews), AI assistants (ChatGPT, Perplexity, Gemini, Claude), and the tools that audit websites.

What Google says about the file and AI Overviews?

If you're asking whether the file helps your SEO in Google, start with Google's own guidance. Google's documentation on AI features and your website lists standard SEO requirements for appearing in AI Overviews and AI Mode: pages must be crawlable, indexable and eligible to show a snippet. The file is not on that list.

Chrome is a different story. Lighthouse now includes an audit for the file, and Chrome's Lighthouse documentation describes it as an emerging convention that gives LLMs and AI agents a machine-readable summary of a site's content. The audit flags server errors when the file can't be fetched, and it marks a missing file (404) as Not Applicable, because the file is optional.

Lighthouse checking the file shows tooling interest in agent-readable websites. It is not evidence that Google Search or AI Overviews use it to rank or cite pages. For what actually earns AI Overview citations, Google points to the same fundamentals as regular search. Keep in mind that "not needed" is not the same as "harmful".

What AI assistants document, and who is actually using the file?

The AI providers' own crawler documentation is where support would show up. OpenAI's bot documentation describes OAI-SearchBot, GPTBot and ChatGPT-User. Perplexity's crawler documentation covers PerplexityBot and Perplexity-User. Both explain their user-agents and how to control them through robots.txt. As of October 2026, none of the crawler documentation from OpenAI, Perplexity or Google says their systems read the file when choosing sources.

Websites using the file are concentrated in developer documentation and AI and infrastructure companies. Two public examples are Cloudflare's developer documentation file (developers.cloudflare.com/llms.txt) and Stripe's documentation file (docs.stripe.com/llms.txt). Both are large technical reference sets where agents genuinely need a map. These are examples of adoption, not proof of results.

How to check whether any bot reads your file?

The only first-party answer to whether the file actually works for you is in your own logs. Run this check once the file has been live for 30 days, and again at 90 days:

1. Filter server or CDN logs for requests to the file's root URL since you published it.

2. Group requests by user-agent, then separate them into four buckets:

• AI retrieval or search bots (OAI-SearchBot, PerplexityBot);

• AI training crawlers (GPTBot);

• SEO audit tools;

• unidentified bots.

3. Verify the bots that claim to be AI crawlers. User-agents are easy to spoof. Check them against the vendor's published IP ranges, or use reverse DNS where the vendor documents it.

A request shows that something fetched the file. It doesn't show that an AI answer cited you, and citation tracking is a separate job.

Other work moves AI visibility far more than the file does. That means crawlable and indexable pages, answer-first page structure, clear entity information backed by schema, and content specific enough to be worth quoting.

How to Create the File in Six Steps?

For a typical B2B site with 10–30 priority pages, building the file is a short editorial task, not a development project. The hard part is choosing what to leave out.

The six steps

1. Pick 10–30 priority pages. Choose core service or product pages, pricing or "how we work", key guides, and about and contact pages. Exclude thin, gated, duplicate, parameterized and noindexed URLs. A curated list is the whole point; listing everything just recreates your sitemap.

2. Write the H1 and a one-to-two-sentence blockquote summary. Say who you are, what you sell and who you serve, in plain language a model can quote accurately.

3. Group links under H2 sections with one-line descriptions. Services, Industries, Guides and Company work for most B2B sites. Each description should say what the page answers, not repeat a slogan. Use canonical URLs that return a 200 status directly, with no redirects.

4. Add an "Optional" section. Put secondary pages here, such as archives, the case-study index and careers, so tools can drop them when context runs short.

5. Publish at the root as plain text or Markdown. Upload the file through your CMS file manager, the web server or a CMS route. Confirm that robots.txt doesn't disallow it and that CDN bot protection doesn't challenge it. Some headless or locked-down CMSs need a developer to add a root-level route.

6. Decide on a full-text companion file, then set an owner and review cadence. Add the full-text file only for documentation-heavy sections. Name an owner who reviews the file after every restructure, migration or major launch, and at least quarterly.

Our practical estimate is that writing the file takes under an hour for a small B2B site once the page list is agreed. Internal approvals usually take longer than the writing. If the file is part of a full AEO workflow, build the page list from the same priority pages you're already optimizing.

An example file for a B2B website

Illustration only: a fictional company, not a client.

The file opens with its one required element, the H1 line # Northwind Analytics, and continues as shown below. Paths are written relative to the site root to keep the example short; in a live file, use full canonical URLs on your own domain.

Northwind Analytics is a fictional US company that helps mid-market manufacturers connect ERP, production and sales data into one reporting platform. This file lists the pages that best explain our services, industries and methods.

Northwind works with B2B manufacturers in North America. We do not sell software licenses; we deliver implementation and managed services.

Services

• [Data platform implementation](/services/data-platform): Scope, phases and typical deliverables for building a manufacturing data platform.

• [ERP data integration](/services/erp-integration): How we connect ERP, MES and CRM data, and which systems we support.

• [Managed reporting](/services/managed-reporting): What ongoing dashboard support includes and how service levels work.

Industries

• [Industrial equipment manufacturers](/industries/industrial-equipment): Common data problems and use cases in equipment manufacturing.

Guides

• [How to plan a data platform project](/guides/data-platform-planning): Step-by-step planning guide with timelines and roles.

• [ERP data quality checklist](/guides/erp-data-quality): Checks to run before migrating or integrating ERP data.

Company

• [About Northwind](/about): Team, history and locations.

• [Contact](/contact): How to reach sales and support.

Optional

• [Case study index](/case-studies): Summaries of published client projects.

• [Blog archive](/blog): All articles by topic.

Every description is factual and specific. Nothing promotional or unverifiable belongs in the file, because a model may repeat it word for word.

llms.txt vs robots.txt vs sitemap.xml

These three root files are often confused, but only one of them controls access.

Blocking or allowing AI crawlers is a robots.txt job. The AI guide file can't grant or deny access, and it doesn't govern training or attribution. Even robots.txt has limits: Google's robots.txt documentation explains that blocking crawling isn't a reliable way to keep a page out of Google's results. If that distinction is new to you, it helps to understand how crawling and indexing differ before changing any of these files.

Google's crawler documentation shows how granular robots.txt can be. The Google-Extended token controls whether your content is used to train and ground Gemini models, but it does not affect Google Search, including AI Overviews. Meanwhile, the sitemaps.org protocol describes sitemap.xml as a URL list for discovery, so it plays no part in access control.

The three files complement each other, and the AI guide file replaces neither of the others. One common mistake is listing a URL in the file that robots.txt disallows. Tools that respect robots.txt simply won't fetch that page.

Should Your Team Create the File?

If you're asking whether you should create the file at all, the short answer is yes when it takes your team under an hour and you have clear priority pages or documentation, but don't expect it to change rankings or citations on its own. The right way to build it depends on your site's size and how often its content changes.

Who benefits most, and who can skip it for now?

Do it now if you run SaaS, developer or API documentation, a large content library, or a B2B site with a clear, stable set of key pages and someone to maintain the file.

Treat it as low priority if your pages aren't reliably crawlable or indexable, lack answer-first structure or have no schema. Fix those first: they are documented requirements, and the file isn't. If you're deciding where AI visibility effort should go, it's worth seeing how AEO and GEO priorities compare.

So, is the file worth it? Here is the cost-benefit, stated plainly:

• Effort: low.

• Risk: low, provided the file is accurate and maintained. Lighthouse, as noted above, treats a missing file as Not Applicable.

• Benefit: unproven as of October 2026.

The file is not mandatory. Multi-brand or multi-language groups should decide per domain and per subdomain, not once for the whole company.

View Our Work

Manual, plugin or generator: choosing a creation method

There are four ways to produce the file. Choosing a generator over manual curation is mainly a trade between control and upkeep.

 Our recommendation: curate by hand if you have under ~50 key pages. Above that, or if content changes often, use a generator or a script, and always follow it with a human edit that removes low-value URLs and rewrites the descriptions. Never publish generator output unedited. It tends to include tag pages, paginated URLs and outdated content.

Before deciding, it helps to know how AI assistants currently describe your brand. Centric AI audit tool is a quick starting point.

How to Test and Validate the File?

Whether you use an llms.txt checker, an llms.txt validator or a manual review, the file should pass these five checks before it ships:

1. Load the file's root URL in a browser. Confirm it returns HTTP 200, not a redirect, a 403 or a bot-challenge page.

2. Confirm the response is plain text or Markdown in UTF-8, with no HTML wrapper added by the CMS.

3. Check that every listed URL resolves directly with a 200, with no 404s, redirect chains, noindexed pages or robots-blocked pages. A redirect and status checker speeds this up on longer files.

4. Check that the structure matches the llmstxt.org proposal: one H1, a blockquote summary, H2 sections, and list items in the form [name](url): description.

5. Run Chrome Lighthouse. Its audit, covered earlier, flags errors when the file can't be retrieved.

A valid file only proves the file is readable. It doesn't prove any AI system uses it; that answer comes from the server log check described earlier.

Best Practices and Mistakes to Avoid

Most problems come from neglect, not from the initial build, so the llms.txt best practices below are mainly about upkeep.

Mistakes to avoid:

• Listing every URL on the site.

• Leaving dead or redirected links after a migration.

• Accidentally blocking the file through robots.txt or a web application firewall.

• Keeping a bloated full-text companion file that nobody updates.

• Publishing unedited generator output.

• Writing promotional claims into descriptions.

• Treating the file as a ranking lever and skipping structural AEO work.

Best practices:

• Use canonical URLs only.

• Write factual, one-line descriptions.

• Name one owner for the file.

• Review it after each restructure or migration, and at least quarterly.

• Keep it consistent with your sitemap and robots.txt.

When to Bring in Expert Help?

This stops being a one-file task in a few situations:

• the site spans many sections, brands, subdomains or languages;

• the CMS is headless or locked down;

• key content is rendered with heavy JavaScript;

• the file is one part of a wider AEO program covering page structure, schema, crawl access and citation tracking.

In these cases, enterprise SEO support helps keep the files, templates and priorities aligned.

Planning llms.txt for online store sites? Point it at a small set of pages:

• top-level categories;

• shipping, returns and warranty policies;

• sizing or buying guides;

• a short list of best-selling products.

Don't list every SKU. Catalogs change too fast for a static file to stay accurate.

Talk to Our Experts Now!

Frequently Asked Questions

Do these files actually work?

They make your key pages easy for an LLM or agent to find. As of October 2026, however, the crawler and search documentation from OpenAI, Perplexity and Google doesn't mention using the file to choose sources, so treat it as low-cost housekeeping rather than a citation lever. Your own server logs, as described in the evidence section, show whether anything fetches it.

Is the file mandatory?

No. It's a voluntary proposal, and even Chrome's Lighthouse treats a missing file as not applicable.

How do I check the file before publishing?

Confirm that the file's root URL returns a 200 status as plain text or Markdown, that every link resolves without redirects, and that the structure follows llmstxt.org. Lighthouse can also flag retrieval errors, as covered in the testing checklist.

How do I generate one?

Write it by hand if you have fewer than ~50 key pages. For larger sites, use a plugin, a generator or a script built from your sitemap or CMS, then edit the output by hand before publishing.

Conclusion

If you came here asking what is llms.txt and whether it's worth the effort, the short version is this. It's a curated Markdown guide at your site's root that points AI systems to your best pages. It is not a standard, not an access control and not a documented ranking factor. It's quick to build and low risk, but its benefit for AI citations is still unproven.

Documentation-heavy and well-structured B2B sites can publish one now and check their logs to see what fetches it. For everyone else, crawlable, well-structured, schema-backed pages come first, because Google documents those requirements for AI features. Centric insights on answer engine optimization cover that higher-impact work in depth.