llms.txt generator: how to create and validate your file

llms.txt is the AI-search equivalent of robots.txt. Where robots.txt tells crawlers what they can and can't access, llms.txt tells AI crawlers what they should prioritize reading. It's a plain text file at your root domain that lists your most important pages with their titles and one-line descriptions.

The standard was proposed to solve a discovery problem: AI crawlers visit millions of pages but have limited context about which ones are most valuable to read deeply. Your sitemap lists every page; your llms.txt highlights the ones that are most worth citing. Think of it as a curated reading list for AI engines.

The format is intentionally simple. The file opens with your brand name (prefixed with #) and an optional tagline (prefixed with >). Then list your pages using Markdown-style links with descriptions. Group pages by section using ## headings.

The fastest approach is to derive it from your sitemap. Pull the URLs from sitemap.xml, add titles and meta descriptions for each, and format them as Markdown links. For sites with hundreds of pages, curate rather than list everything — llms.txt should highlight your 20–50 most important, citation-ready pages, not replicate your entire sitemap.

For the companion llms-full.txt file (which includes the full text of each page for AI deep reading), you'll need to strip HTML and output clean text. This file can be larger — 500KB or more for a substantial content library — and gives AI engines the full context without requiring them to crawl and parse each page individually.

After deploying, validate by fetching the live URL (curl https://yoursite.com/llms.txt) and checking that it returns plain text with the correct format. Common issues: the file returns HTML instead of text/plain (fix your Content-Type header), the file is empty or cached stale (check your deployment), or links are broken (verify each URL resolves).

Maintain it as part of your deploy process. When you publish new content, add it to llms.txt. When you remove or unpublish content, remove it. Fastlook generates and updates llms.txt automatically as part of the publishing workflow.