llms.txt: How to Make Your Website Readable by AI Agents

What llms.txt is, why it matters for agent-native websites, and how to write one that helps AI systems find the right content without guessing.

If you want AI agents to understand your website, the first problem is not intelligence. It is navigation.

Large language models can read a lot of text, but they are still bad at guessing which pages matter, which pages are current, and which pages are safe to rely on. A site can be perfectly usable for humans and still be awkward for agents because the important information is buried behind menus, scripts, search filters, or sprawling documentation trees.

That is the problem llms.txt tries to solve.

What is llms.txt?

llms.txt is a proposed plain-text file for websites, typically placed at the root of a domain as /llms.txt. The idea is simple: give AI systems a concise map of the site’s most useful, human-written entry points.

Think of it as a curated guide for machines. Not a crawler directive. Not a full content dump. A guide.

The file is written in Markdown-like plain text, so it is easy for humans to edit and easy for agents to parse. In practice, most implementations keep it to a single page of links and short descriptions rather than a long narrative. It usually includes:

a title
a short summary of the site
one or more sections with links to key pages
optional notes for agents about what the site is for

The project is associated with the broader effort at llmstxt.org, which describes the format and its intended use.

Why it matters for agent-native sites

For an agent, the web is not just a set of pages. It is a task environment.

An agent trying to compare pricing, read docs, or complete a workflow needs fast access to the right source material. If the only path is through a homepage hero, a JavaScript-heavy navigation menu, and a search box with no obvious query terms, the agent has to spend tokens and time figuring out what humans already know.

llms.txt helps by making the site legible at the top level.

That matters most when the site has several distinct content types that agents need to distinguish quickly:

product documentation with setup steps and API references
pricing pages with plan limits, usage caps, and billing terms
policy pages with terms, privacy, security, and acceptable-use rules
support content with troubleshooting, status, and contact options
onboarding flows with account creation, verification, and first-run steps

It also matters for founders and developers because it gives you some editorial control. Instead of leaving agents to infer importance from internal links, you can say: start here, then read this, then use that.

The spec format, in practice

The format is intentionally lightweight. A common structure looks like this:

# Site Name

Short description of what this site does and who it is for.

## Section Name
- [Page title](https://example.com/page): one-line description
- [Another page](https://example.com/another): one-line description

## Another Section
- [Doc](https://example.com/doc): one-line description

A few practical rules:

Use Markdown headings.
Link to canonical pages.
Keep descriptions short and specific.
Prefer a few high-value links over a giant index.
Avoid marketing language. Agents do better with plain descriptions than slogans.
Put the most actionable pages first: getting started, docs, API reference, pricing, policies, and support.

There is no reason to include every blog post, tag page, or archive. If the file becomes a sitemap clone, it stops being useful.

A real production example

A useful production example is Vercel’s llms.txt. Vercel is a good case because its site spans product pages, docs, deployment guides, and platform information. Rather than forcing an agent to discover everything from scratch, the file points to the most relevant entry points, including documentation, platform pages, and other canonical resources.

Another public example is Anthropic’s llms.txt. It shows the same basic pattern: a compact overview followed by links to the pages most likely to matter for a model or agent, such as product information, docs, and policy pages.

These examples are useful not because they are perfect, but because they show how a production company can make a site more navigable without redesigning the whole information architecture.

How to write one for your site

Start by asking what an agent would actually try to do on your site.

For a software company, the answer is usually something like:

understand the product
read docs
inspect API references
check pricing
review terms or security policies
find onboarding instructions

Then build the file around those tasks.

A good llms.txt often includes sections like:

Overview — the homepage or product summary
Docs — getting started, tutorials, API reference
Pricing — pricing and plan limits
Policies — terms, privacy, security, acceptable use
Support — help center, status page, contact channels

Example:

# Acme AI

Acme AI provides tools for automating customer support workflows.

## Start here
- [Homepage](https://acme.ai): Product overview and positioning
- [Getting Started](https://acme.ai/docs/getting-started): Setup guide for new users

## Documentation
- [API Reference](https://acme.ai/docs/api): Endpoints, auth, and request examples
- [Webhooks](https://acme.ai/docs/webhooks): Event delivery and retry behavior

## Business
- [Pricing](https://acme.ai/pricing): Plan tiers and usage limits
- [Security](https://acme.ai/security): Security practices and compliance

This is enough to be useful. It is also easy to maintain because each link maps to a stable page that already exists for humans.

The contrarian view: do you actually need one?

Not every website needs llms.txt.

If your site is tiny, static, and already has a clean information structure, the file may not change much. If your important content is already in a few obvious pages with solid internal linking, agents may do fine without it.

There is also a risk of treating llms.txt like a magic compatibility layer. It is not. It does not guarantee that every model will use it. It does not replace good HTML, structured data, or accessible navigation. And it will not fix content that is fragmented, outdated, or locked behind unnecessary friction.

The most honest way to think about it is this: llms.txt is helpful when your site has enough surface area that a curated map saves time.

How it fits with robots.txt and sitemap.xml

These files do different jobs.

robots.txt tells crawlers what they may or may not fetch.
sitemap.xml lists URLs you want search engines to know about.
llms.txt highlights the pages you want AI systems to read first.

They are complementary. A site can have all three.

If you are building an agent-native site, the combination is stronger than any one file alone. robots.txt manages access, sitemap.xml provides coverage, and llms.txt provides intent.

Implementation tips

A few small choices make a big difference:

Put the file at /llms.txt
Serve it as plain text or Markdown
Keep URLs stable and canonical
Update it when docs or product structure changes
Avoid duplicating low-value pages
Write for clarity, not discoverability tricks
Include only pages that a human would confidently send a customer to

If you use a framework like Next.js, adding a static root file is straightforward. If your site is on a platform like Vercel, deployment is also easy because the file can live alongside your other static assets.

The Bottom Line

llms.txt is a simple idea with a practical goal: help AI agents find the right parts of your website without guessing.

It is not a standard in the same sense as robots.txt or sitemap.xml, and it will not matter equally for every site. But for documentation-heavy products, SaaS companies, and any site that expects agents to browse, compare, or act on content, it is a low-cost way to make your site easier to understand.

If your website already has a clear structure, llms.txt may be a small improvement. If your site is large or complex, it can be the difference between an agent getting lost and an agent getting to work.