llms.txt Explained: Why Your Site Needs One (+ Example)

Somewhere in the last two years, a second audience showed up for your website. It doesn't scroll, it doesn't click ads, and it doesn't care what your hero section looks like. It's an AI agent — ChatGPT browsing the web, Perplexity answering a question, Claude researching a topic, an autonomous coding assistant checking your docs — and it's reading your site to answer someone else's question, usually in a few seconds, with a token budget that runs out fast.

That audience is why llms.txt exists. It's a small, plain-text file that's quietly becoming a best practice the same way robots.txt and sitemap.xml did — not because a search engine mandated it, but because it solves a real problem cheaply. This post covers what it actually is, why it's worth an hour of your time, how to write one from scratch, and how to wire it into a real website — including the one you're reading this on.

What llms.txt actually is

llms.txt is a plain Markdown file you publish at https://yoursite.com/llms.txt. It's not a rendering directive like robots.txt (it doesn't allow or block anything) and it's not a URL manifest like sitemap.xml (it doesn't try to list every page). It's closer to a README for your entire website — a short, curated brief that tells a language model what your site is, what matters on it, and where to find the details.

The format was proposed by Jeremy Howard (Answer.AI) in September 2024 and lives at llmstxt.org. The core idea is simple: web pages are built for humans and browsers — full of navigation bars, cookie banners, ads, and JavaScript that a language model has to wade through just to extract a few facts. llms.txt skips all of that and hands the model a clean, link-dense index it can parse in one pass.

Most llms.txt implementations ship alongside a companion file, llms-full.txt, which inlines the actual content of the important pages into a single document. Think of it as the difference between a table of contents and the whole book bound as one file — the first helps an agent decide what's relevant, the second lets it read everything without firing off another ten requests.

Why it actually matters

Here's the honest case for it, without the hype.

Context windows are a real constraint, and noisy pages waste them. When an agent fetches a normal page, it's pulling down your entire DOM — nav, footer, cookie banner, tracking scripts, marketing copy — and has to figure out what's signal. A well-written llms.txt is pure signal: what you make, what it does, and where to read more. That's a better use of a limited context budget for the agent, and it means the summary it gives a user about you is more likely to be accurate.

More answers are coming from agents than search results pages. People increasingly ask ChatGPT, Perplexity, or Claude a question instead of typing it into Google and clicking a blue link. Whether or not any given platform is confirmed to fetch llms.txt today, the direction is obvious: a growing share of "traffic" to your site is really an AI system reading it on someone's behalf. Making that reading easier is a low-cost bet on where things are headed.

Your existing SEO signals weren't built for this. robots.txt tells crawlers what they may fetch. sitemap.xml tells them what exists. Schema.org markup tells them what a page is, in a machine-readable way. None of them tell an agent, in plain language, what your product does and why it matters — which is exactly the gap a curated, human-readable brief fills.

It costs almost nothing to do. This is the real argument. Writing a good llms.txt takes an hour, it's a static text file with zero moving parts, and it can't break your site, hurt your rankings, or conflict with anything else you're already doing.

llms.txt vs. the files you already have

They're not competing standards — they're different jobs. Keep all of them. llms.txt is additive.

The anatomy of an llms.txt file

The spec is deliberately minimal, which is most of why it's easy to adopt:

1. An H1 — your site or project name. Required, one line.
2. A blockquote immediately after the H1 — a one- or two-sentence summary of what the site/company is. Required.
3. Optional free paragraphs — extra context, without headings, if the summary needs a bit more room to breathe.
4. H2 sections — logical groupings of links, each a Markdown list: - [Title](URL): short description.
5. An optional "Optional" H2 — this one is spec-defined, not just a name. It signals links an agent can skip first if it's working with a tight context budget.

Here's the bare skeleton:

# Your Company Name
 
> One or two sentences. What you make, who it's for, what makes it different.
> This is the part a model will quote back almost verbatim, so make it count.
 
Optional paragraph with a bit more context — principles, positioning,
anything that helps a model represent you accurately.
 
## Core pages
 
- [Home](https://example.com/): What this company does, in one line.
- [Product](https://example.com/product): What it does and who it's for.
- [Pricing](https://example.com/pricing): How much it costs and how it's billed.
 
## Docs
 
- [Getting Started](https://example.com/docs/start): Setup in under 5 minutes.
- [API Reference](https://example.com/docs/api): Full endpoint reference.
 
## Optional
 
- [Blog](https://example.com/blog): Articles and product updates.
- [Careers](https://example.com/careers): Open roles.

That's genuinely most of the spec. The discipline is in what you leave out.

A real example

Here's an excerpt from the actual llms.txt this site ships at degird.com/llms.txt, trimmed for length:

# Degird
 
> Degird is a multi-product technology company that builds purpose-built software
> across SaaS, desktop, WordPress, and browser extensions — plus an AI-powered
> agency for app development, video production, and motion graphics. Every
> product is built in-house, privacy-first, and priced without subscription traps.
 
## Core pages
 
- [Home](https://degird.com/): Company overview, product ecosystem, and services.
- [Products](https://degird.com/products): Full product portfolio across all four categories.
 
## SaaS products
 
- [Dormefy](https://degird.com/products/dormefy): Dormitory and mess meal management
  application — create, manage, and publish a dorm in minutes.
 
## WordPress solutions
 
- [SyncDock](https://degird.com/products/syncdock): Headless WordPress publishing API
  for AI agents and automation — hashed API keys, HMAC-signed requests, granular
  scopes, and Gutenberg block support.
 
## Optional
 
- [Blog](https://degird.com/blog): Product updates, engineering write-ups, tutorials,
  and industry insights.
- [Full context for LLMs](https://degird.com/llms-full.txt): Expanded, single-file
  description of every product and service.

Notice what it's not doing: no adjectives doing the heavy lifting ("revolutionary," "industry-leading"), no duplicated nav links, no marketing paragraph before the summary. Every line either states a fact or points somewhere useful.

How to write your own, step by step

1. Inventory your important pages — and stop at 15–30. Homepage, product or pricing pages, docs entry points, your two or three most useful blog posts, about/contact. If a page wouldn't make it into a five-minute pitch about your company, it doesn't belong here. That's what sitemap.xml is for.

2. Write flat, factual, one-line descriptions. Say what a page is and what it does, not how great it is. "Headless publishing API for AI agents and automation" is useful to a model. "The most powerful publishing solution on the market" is not — it's a claim with nothing to retrieve.

3. Group pages into a small number of H2 categories. Three to six sections is normal. Categories should map to how someone would actually think about your site, not your internal org chart.

4. Write the H1 and the blockquote last. This is the part most likely to get quoted directly, so draft it after you've written everything else and you actually know what your site is about that day.

5. Use the Optional section on purpose. Put your blog, careers page, and other nice-to-haves there. It's a real signal in the spec, not just a label — agents with a tight budget are meant to skip it first.

6. Build llms-full.txt if you have the content to justify it. Take the pages you linked in the "core" sections and inline their actual content — not a copy-paste of the raw HTML, a clean plain-text version — into one file. Skip it if your site is small; a good llms.txt alone is enough.

How to implement it on your site

The good news: there's no framework requirement and nothing to configure. llms.txt is just a static text file at your domain root.

Static file — the right default for almost everyone. Drop llms.txt (and llms-full.txt, if you have one) straight into your public/static assets folder. In a Next.js App Router project, that's literally public/llms.txt — no route, no build step, nothing else to wire up. That's exactly how this file is served on this site.

public/
├── llms.txt
├── llms-full.txt
├── favicon.ico
└── ...

Push it, and it's live at yoursite.com/llms.txt on the next deploy. For a WordPress site, it's the same idea: upload the file to your web root next to wp-config.php. For any static host — Netlify, Vercel, GitHub Pages — it works exactly like robots.txt does today, because it's served the same way.

Generated at build time — worth it once your site outgrows a hand-maintained list. If you're publishing content often enough that a static file will drift out of date, generate it from the same data your CMS or blog index already has. In a Next.js App Router project, that's a route handler:

// app/llms.txt/route.ts
import { getAllPosts } from "@/lib/blog";
 
export async function GET() {
  const posts = getAllPosts().slice(0, 10);
 
  const body = `# Your Company
 
> One or two sentences describing what you do.
 
## Recent posts
 
${posts
  .map((p) => `- [${p.title}](https://yoursite.com${p.url}): ${p.description}`)
  .join("\n")}
`;
 
  return new Response(body, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

Now your llms.txt updates itself every time you publish, the same way sitemap.xml already does on most modern frameworks. Whichever route you take, the file is inert — it's just text. There's no server logic to secure and nothing that can take your site down.

Best practices for agent browsing

A handful of rules separate an llms.txt that actually helps from one that's just a checkbox:

• Optimize for tokens, not completeness. Every line costs an agent context budget. Ruthlessly cut anything that isn't a top-priority page.
• Write literal, factual descriptions. Agents (and the people reading what they generate) weight concrete claims over adjectives. "One-time $19 purchase, runs offline" beats "affordable and powerful."
• Use absolute URLs, always. The file gets fetched and cached independently of the page it lives on — relative links break the moment it's read out of context.
• Keep it current. A stale llms.txt linking to a page you removed is worse than no file at all — it actively misleads whatever reads it. Update it the same cadence you'd update a sitemap.
• Use the Optional section deliberately, not as a dumping ground. It's the one place in the spec that explicitly helps an agent triage under a tight budget.
• Don't duplicate your sitemap. If llms.txt starts looking like an exhaustive URL list, you've lost the plot — that job already belongs to sitemap.xml.
• Pair it with real fundamentals. Genuine page content, correct meta descriptions, working structured data, and a clean robots.txt still matter more than this file ever will. llms.txt is the easy 5% on top of a site that's already doing the hard 95% right.

Frequently asked questions

Does llms.txt guarantee ChatGPT or Perplexity will cite my site?

No. No major AI platform has publicly confirmed it automatically crawls llms.txt as of today. It's a low-cost bet on a direction the ecosystem is clearly moving in — and the exercise of writing one forces you to describe your own site clearly, which is useful on its own regardless of who reads the file.

Does llms.txt replace robots.txt or sitemap.xml?

No. They do different jobs — permissions, exhaustive URL listing, and curated summary, respectively — and you should keep all three.

Will adding llms.txt hurt my regular Google SEO?

No. It's a plain text file with no directives. Standard search crawlers simply ignore a file they don't recognize; it can't conflict with anything else on your site.

How often should I update it?

Whenever you'd update a README — new flagship product, major repositioning, or a page you linked getting retired. It doesn't need daily attention, but it shouldn't go stale for a year either.

Is there a required file size limit?

The spec doesn't set a hard cap, but the entire point is brevity. If your llms.txt runs past a couple hundred lines, you're probably listing pages instead of curating them — move the extra detail into llms-full.txt instead.

The takeaway

llms.txt isn't going to make or break your search rankings, and anyone who tells you it will is overselling a one-page text file. What it is: a genuinely useful, close-to-free way to hand AI agents a clean, accurate summary of your site instead of making them guess from a rendered page full of nav bars and cookie banners. Write it in an hour, keep it honest, keep it current, and put it next to robots.txt and sitemap.xml where it belongs — one more small signal in a stack that, together, actually adds up.

You can see the live version of everything in this post at degird.com/llms.txt and the expanded degird.com/llms-full.txt. If you're publishing content for AI agents to consume beyond a static file — pushing updates from a headless CMS, syncing docs automatically, or building the kind of publishing pipeline this post describes — that's exactly what SyncDock does for WordPress, and it pairs well with what we covered in Headless WordPress for AI Publishing.