The llms.txt file is a proposed way to give AI systems a cleaner, curated map of the content on your website. As a GEO consultant, it is one of the questions clients bring me most often right now.
Think of it as a short Markdown guide written for large language models and AI agents. Instead of making an AI system interpret navigation, scripts, sidebars, ads, and hundreds or thousands of HTML pages, an llms.txt file can point it toward the resources that matter and explain what those resources contain.
That sounds useful as websites compete for visibility in ChatGPT, Claude, Perplexity, Gemini, and other forms of AI-powered search.
There is also an important catch.
As of 2026, no major AI provider has publicly confirmed that publishing an llms.txt file gives a website a ranking advantage or that the file is a required retrieval input. OpenAI, for example, tells publishers that allowing its search crawler through robots.txt helps content become discoverable in ChatGPT search. That is a documented mechanism. llms.txt does not currently have the same level of provider confirmation.
So should you create one?
For many websites, yes, but treat it as a low-cost technical experiment rather than a magic switch for AI visibility.
What Is an llms.txt File?
An llms.txt file is a Markdown document designed to help AI models and AI assistants understand the important content available on a website.
Jeremy Howard of Answer.AI introduced the proposal in 2024. The llms.txt specification has since evolved, including a v2 update in August 2026.
The idea addresses a legitimate problem: websites are designed primarily for humans and browsers. An ordinary webpage can contain navigation, JavaScript, advertisements, footers, related posts, tracking code, and other elements surrounding the information someone actually came to find.
AI agents do not necessarily need all of that.
An llms.txt file provides a simpler layer containing a description of the site and curated links to useful resources. The recommended links can lead to Markdown files or other resources that are easier for machines to consume.
The current specification allows an llms.txt file at /llms.txt or within a subpath such as /docs/llms.txt. A file within a subpath applies to content beneath that path, while placing one in the root directory remains the simplest sitewide implementation.
What Goes in an llms.txt File?
The format is intentionally simple.
According to the llms.txt specification published by Jeremy Howard at Answer.AI, the file uses Markdown and follows a basic hierarchy:
- An H1 containing the project or website name
- A blockquote containing a concise summary
- Optional additional context about the project
- H2 sections containing lists of annotated links
Only the H1 is strictly required by the specification, but a useful llms.txt file should provide enough context and organization to make the resource worthwhile.
A simplified version might look like this:
# Example Company
> Example Company provides accounting software for small businesses.
Important information about Example Company and its products.
## Product Documentation
- [Getting Started](https://example.com/docs/getting-started.md): Setup instructions for new accounts.
- [Features](https://example.com/docs/features.md): Overview of the platform's primary features.
## Resources
- [Pricing](https://example.com/pricing.md): Current plans and product pricing.
- [Support](https://example.com/support.md): Customer support information.This is fundamentally different from giving an AI crawler a giant list of URLs. You are curating important resources and explaining why each one matters.
That distinction is central to understanding llms.txt.
llms.txt vs. robots.txt
The easiest mistake is assuming an llms.txt file is basically robots.txt for AI crawlers.
It isn’t.
A robots.txt file controls crawler permissions. It tells compliant crawlers which areas of a website they can or cannot access.
For example, OpenAI explicitly documents robots.txt controls for its crawlers. Anthropic likewise states that its crawlers respect robots.txt instructions.
An llms.txt file performs a different job.
robots.txt answers: “Are you allowed to crawl this?”
llms.txt answers: “If you’re trying to understand this website, here are the resources I recommend.”
One controls access. The other provides curation and context.
That means adding an llms.txt file does not fix an AI crawler blocked by robots.txt. If GPTBot, a search crawler, or another bot cannot access your content, adding another text file isn’t going to sneak it past the bouncer.
Crawler accessibility still matters.
llms.txt vs. sitemap.xml
A sitemap has another distinct job.
Your sitemap.xml typically helps search engines and crawlers discover URLs on your website. Depending on the site, that sitemap might contain hundreds, thousands, or millions of URLs.
An llms.txt file is supposed to be selective.
Rather than saying, “Here are all my indexable URLs,” you can effectively say, “Here are the resources most useful for understanding this topic, product, or website.”
That matters because AI models have finite context windows. Dumping an entire website into a model is not necessarily efficient or useful.
The llms.txt proposal therefore complements a sitemap instead of replacing it.
For technical SEO, keep your sitemap.xml.
For crawler permissions, maintain robots.txt.
For curated machine-readable context, consider llms.txt.
They solve three different problems.
What Is llms-full.txt?
You may also encounter llms-full.txt.
The distinction is relatively straightforward:
- llms.txt acts primarily as a concise guide or index pointing AI agents toward important resources.
- llms-full.txt provides a much larger single-file version containing the detailed content itself.
The llms-full.txt approach can be useful when you want an AI system to retrieve substantial documentation without visiting many separate resources.
That convenience comes with a tradeoff.
A large llms-full.txt file can consume far more of an AI model’s context window. For a substantial website, creating one enormous file containing everything may defeat the purpose of efficient context retrieval.
In most cases, I would start with a focused llms.txt file. Consider llms-full.txt when your documentation or content set is compact enough that consolidating it produces a genuinely useful resource.
Does llms.txt Actually Improve AI Visibility?
This is where the marketing gets ahead of the evidence.
There are good theoretical reasons an llms.txt file could help AI agents consume a website. The format reduces noise, provides context, and points machines toward authoritative resources.
Adoption is also real. The llms.txt project reports that thousands of sites now publish the file, and platforms including Mintlify automatically generate it. The specification also lists integrations for CMS and documentation platforms.
But adoption is not the same thing as ranking evidence.
As of 2026, there is no public confirmation from a major AI provider that an llms.txt file directly improves rankings, citations, or retrieval in ChatGPT, Claude, Gemini, or Perplexity.
That distinction should shape your AEO and GEO strategy.
If someone promises that adding llms.txt will immediately increase AI visibility, ask them for evidence.
The more defensible position is that llms.txt is a low-cost method of making important information easier for compatible AI agents to locate and interpret. Whether that translates into measurable gains from AI-powered search remains something website owners should test.
Where Does llms.txt Fit Into SEO, AEO, and GEO?
Traditional search engine optimization remains foundational.
Good technical SEO makes your site crawlable. Strong information architecture helps machines understand relationships between pages. Useful content satisfies search intent. Internal links establish context and hierarchy.
None of that becomes obsolete because you created an llms.txt file.
AEO, or answer engine optimization, expands the focus toward making information easy for systems to extract and use when answering questions.
GEO, or generative engine optimization, focuses more specifically on visibility within generative AI experiences. If the term is new to you, start with what generative engine optimization actually is.
An llms.txt file fits more naturally into AEO and GEO than conventional SEO because it is explicitly designed to make web resources easier for AI systems to consume. I have written a fuller breakdown of GEO vs SEO and which parts of the old playbook still apply.
Still, your broader SEO strategy should come first.
If your website has serious crawling problems, duplicate content, weak information architecture, poor content, or broken internal links, fixing those issues probably deserves priority over building an llms.txt file. That is the kind of work a technical SEO consultant should handle before you spend time on machine-readable extras.
How to Create an llms.txt File
You have three practical options.
1. Create It Manually
For a small website, manual creation is often easiest.
Create a plain text file named llms.txt using Markdown formatting. Add your site name, summary, any necessary explanatory context, and H2 sections containing annotated links to your most important resources.
Then publish it so it is publicly accessible, typically at:
https://example.com/llms.txt
You can also use a subpath-specific implementation such as /docs/llms.txt for a particular section of a site.
For static sites, this may be as simple as placing the file in the public root directory during your build process and deploying the site.
The most important part is not the technical complexity. It is deciding which resources deserve inclusion.
2. Generate llms.txt in WordPress
WordPress site owners may not need to maintain the file manually.
The current llms.txt project documentation lists WordPress plugins including Yoast SEO and AIOSEO as integrations capable of generating or maintaining an llms.txt file.
If your existing SEO plugin supports the feature, start there rather than installing another plugin solely to generate one small Markdown document.
After enabling it, verify that the file loads publicly and review its contents. Automation is useful, but an automatically generated list is not necessarily the best curated representation of your website.
3. Use an llms.txt Generator
An llms.txt generator can crawl or inspect a site and produce a starting file automatically.
Tools and platforms in the ecosystem include Firecrawl and documentation systems such as Mintlify. Developers can also generate the file during a static-site build or use an MCP server that exposes llms.txt resources to compatible agents.
There are community directories such as directory.llmstxt.cloud as well.
The principle is the same whether you’re using a generator, Zapier workflow, plugin, or custom script: review the output before publishing it.
Do not assume more URLs equal a better llms.txt file.
Curation is the feature.
Should You Create Markdown Versions of Your Pages?
The proposal goes beyond a single llms.txt file.
It recommends making important content available as Markdown files that AI systems can consume without parsing the full presentation layer of HTML pages.
The v2 proposal supports Markdown alternatives using forms such as page.html.md or page.md. It also recommends standard link relationships that can identify the Markdown alternative and the applicable llms.txt resource.
That can make particular sense for:
- Developer documentation
- API references
- Knowledge bases
- Product documentation
- Technical tutorials
- Structured reference content
For a five-page local service website, producing parallel Markdown files for everything is probably overkill.
For a documentation portal containing thousands of pages used regularly by chatbots and coding agents, the equation changes.
How to Measure Whether llms.txt Is Working
Do not measure success by checking whether /llms.txt returns a 200 status code and calling the project finished.
Measure outcomes.
Monitor server logs to determine whether AI crawlers request the file. Look for changes in referral traffic from AI assistants. Track whether your brand and pages appear in relevant AI responses. Compare citation frequency before and after implementation where you have a repeatable monitoring process.
You can also use AI-oriented crawling tools to validate that the file is accessible and parseable.
The difficult part is attribution.
Even if AI visibility increases after publishing llms.txt, that does not prove the file caused the improvement. Content updates, increased brand mentions, links, crawler changes, model updates, and other factors can all affect visibility.
Treat implementation like an experiment rather than a ranking hack.
Who Should Bother With llms.txt?
An llms.txt file makes the most sense for sites where machines benefit from a curated path through substantial information.
I would prioritize it for:
- SaaS companies with extensive product documentation
- Developer platforms and API documentation
- Large knowledge bases
- Publishers with substantial evergreen reference content
- Companies actively investing in AEO and generative engine optimization
- Sites already monitoring traffic and citations from AI models
The case is weaker for tiny brochure sites, sites with little useful indexable content, or businesses that still have major technical SEO and content problems to solve.
If implementing llms.txt takes 30 minutes because your CMS supports it, the downside is minimal.
If implementing it requires a multi-week engineering project, I would want considerably stronger evidence that AI agents relevant to your business are actually using it.
That is the practical verdict in 2026:
llms.txt is a sensible, low-risk experiment with an unconfirmed payoff. If you would rather have a GEO agency run that experiment alongside the rest of your AI visibility work, that is a reasonable way to handle it too.
It is not mandatory. It does not replace robots.txt. It does not replace your sitemap. And there is currently insufficient public evidence to treat it as a direct ranking factor for AI search.
Make your website accessible, useful, authoritative, and technically sound first.
Then, if the implementation cost is low, give machines a cleaner map.
Frequently Asked Questions About llms.txt
What is an llms.txt file?
An llms.txt file is a Markdown document that provides AI agents and large language models with a curated description of a website and links to important resources. The llms.txt specification defines a simple structure based on headings, explanatory text, and annotated links.
Is llms.txt actually used?
Yes, the format has real adoption. The official project reports thousands of implementations and integrations with several publishing platforms. However, there is no public confirmation from major AI providers that llms.txt is a ranking signal or required retrieval input. Those are two very different claims.
Is llms.txt mandatory?
No. An llms.txt file is a proposed web standard, not a requirement for conventional search engine optimization or inclusion in major AI systems. A website can be crawled, indexed, ranked, and potentially surfaced by AI-powered search without one.
How do I create and implement an llms.txt file for my website?
Create a Markdown-formatted file containing an H1 project name, ideally a blockquote summary and useful context, followed by H2 sections containing annotated links. Publish the llms.txt file at your root directory for a simple sitewide implementation, or use a path-specific file where appropriate. WordPress users can also use supported plugins, while static sites can add the file directly to their public build directory. An llms.txt generator can provide a useful starting point.
What is the difference between llms.txt and robots.txt?
robots.txt manages crawler permissions. llms.txt curates information for AI systems. If you want to prevent or permit AI crawlers from accessing certain areas, robots.txt is the relevant mechanism. If you want to provide AI agents with a curated map of useful resources, that is the role proposed for llms.txt.
What is llms-full.txt and why is it useful?
llms-full.txt is an expanded version intended to consolidate detailed content into a single resource rather than primarily linking to separate documents. It may simplify retrieval for smaller documentation sets, although very large llms-full.txt files can consume substantial context windows.
How can llms.txt impact search engine optimization?
There is currently no confirmed direct SEO ranking benefit from publishing an llms.txt file. Its potential value is primarily associated with AEO, AI visibility, and making content easier for compatible AI models and agents to consume. Treat it as an additional machine-readable layer, not a replacement for conventional search engine optimization.