AI sitemaps and content discovery

How AI engines find your content

Discovery for AI engines works much like for search engines - their crawlers find content through XML sitemaps (which list your URLs), internal links (which lead crawlers from page to page), and by respecting your robots rules. There's no separate mandatory 'AI sitemap' format; the standard discovery infrastructure serves AI crawlers too. Getting the fundamentals right is what makes your content discoverable by AI.

The XML sitemap is the foundation

A complete, current XML sitemap listing all your citable URLs is the discovery foundation. It tells crawlers exactly what content exists and when it was last updated. Keep it complete (every published page), current (accurate lastmod dates), and referenced in robots.txt. A missing or stale sitemap means content may go undiscovered - the simplest, highest-leverage discovery fix.

Internal links + crawler access

Two more pillars: internal linking and crawler access. Strong internal links help crawlers discover pages and understand how content relates (reinforcing topical clusters). And robots.txt must allow AI crawlers (GPTBot, PerplexityBot, etc.) - if you block them, they can't discover you no matter how good your sitemap is. Check both: links lead crawlers through your site, robots lets them in.

• Internal links: connect content so crawlers can traverse and relate it.
• robots.txt: allow AI crawlers or they can't discover you.
• Both work with the sitemap, not instead of it.

Where llms.txt fits

llms.txt is an optional additional signal that curates your best pages for AI - useful as a forward-looking supplement, but it is not a required 'AI sitemap' and doesn't replace the standard XML sitemap. The reliable discovery stack is: complete XML sitemap + strong internal links + AI-crawler access in robots.txt, with llms.txt as a low-cost extra. Don't skip the fundamentals in favor of the emerging signal.