Definition
An XML sitemap is a file, defined by the sitemaps.org protocol, that lists the URLs a site wants search engines to crawl and index, optionally with the date each page last changed. It tells crawlers which pages exist and which have been updated, which matters most for large sites, new pages and pages with few internal links.
The format is simple. Each url entry has a loc (the absolute URL) and an optional lastmod date. A single file can hold up to 50,000 URLs or 50 MB uncompressed, and larger sites split their URLs across several files referenced from a sitemap index; Bing notes an index can reference up to 50,000 child sitemaps. Files must be UTF-8 encoded. You make a sitemap known by submitting it in Google Search Console or Bing Webmaster Tools, or with a Sitemap: line in robots.txt, which any crawler can read.
Only one optional tag still matters. Google's sitemap documentation says it ignores priority and changefreq, and uses lastmod only when it is consistently and verifiably accurate. A significant change, such as edited main content, structured data or links, should update lastmod; a new copyright year should not. Bing gives the same advice in its July 2025 guidance on sitemaps for AI-powered search: it ignores changefreq and priority, prefers lastmod with both date and time in ISO 8601 format, fetches a submitted sitemap immediately and rereads it at least daily, and uses lastmod to decide what to recrawl for both search results and Copilot answers. Setting every lastmod to the time the file was generated destroys the signal.
For AI search, sitemaps work mostly through the big search indexes. Google's AI Overviews and AI Mode are grounded in Google's index, and Copilot in Bing's, so faster and more accurate indexing there feeds AI answers too. Bing recommends pairing sitemaps with IndexNow to notify it of individual changes as they happen. For independent AI crawlers, the picture is less documented: OpenAI's crawler documentation, for example, does not say whether OAI-SearchBot reads sitemaps. Your AI crawler logs answer the question for your own site: check whether each bot requests sitemap.xml and how soon after publishing it fetches new URLs.
Good practice is consistent across engines. List only canonical URLs that return 200 and are indexable, and drop redirects, noindexed pages and duplicates (canonical URLs). Generate the file from the CMS so it stays current, set lastmod from the real content change date, and split large sites by section so Search Console reports coverage per group. Then compare three lists: URLs in the sitemap, URLs crawlers actually fetch, and URLs that get cited. Gaps between them point to crawl budget, quality or content freshness problems, and the AI crawler URL inspector shows what an AI crawler gets from any one of those URLs. An llms.txt file is sometimes proposed as a sitemap for AI, but our analysis found llms.txt has no measurable impact on AI search, so the XML sitemap remains the file to get right.
Examples of XML Sitemaps
- A news site's CMS rewrites lastmod to the build time for every URL whenever the sitemap regenerates. Because those dates are not verifiably accurate, Google may ignore them, so the team switches lastmod to each article's last substantive edit.
- An online store with 400,000 product URLs splits them into nine category sitemaps under one index file, so Search Console shows indexing coverage for each category separately.
- A SaaS company checks its crawler logs and sees Bingbot fetching sitemap.xml daily while OAI-SearchBot never requests it, so it relies on internal links from frequently crawled pages to surface new docs for ChatGPT search.
- After an audit, a publisher removes 12,000 redirected and noindexed URLs from its sitemaps, leaving only canonical pages that return 200.
