Sitemap pages load
Do the pages your sitemap lists actually load, with readable text?
What we look for
We take up to ten pages from your sitemap and fetch each one. Your two newest posts come first, when the sitemap lists any: by lastmod, or without one by the date in a post’s address, or by your feed’s order. Then we prefer pages near the top of the site and ones changed recently, two from a section and one from a numbered series before any more of either, so the sample spreads across the site; a site whose pages all sit in one section still gets ten. No one kind of page fills the sample when others exist: at most half of the ten come from one section of the site (a year in the address, as in /2026/11/16/a-story, counts as one section), and past that the places go to your other sections, your home page’s links to them among them, before any more of the one. A lastmod we would ourselves call a generator’s stamp never picks a post as your newest, and a landing page your navigation lists is never taken for a post because its lastmod is recent. If the sitemap is an index of other sitemaps, we follow it (and any index inside it), asking for up to four sitemaps in all, and spending them where the dates are: one named for posts, articles, a blog, or news first, then the ones the index dates most recently; where the index gives no dates, pages come next, then the rest, with collections and products last. The sitemaps your robots.txt declares are read in the order it declares them, one named for posts still first: you have said which comes first. Sitemaps of images, videos, or attachments are not pages, and we leave them alone. An address that turns out to serve an image or a file rather than a page isn’t a failing page: we pass over it and take the next. So is a page that ends at a login, a form and little else, or a thank-you page, and a second copy of a page we already have (another page of the same list, another region’s edition): each is sampled only when nothing else is left. The same site gets the same sample every scan, unless its sitemap or its feed changes.
A page passes if it loads: it answers normally (a 200) and is a web page, plain text, or Markdown. How much text it has is the content criterion’s question, so a thin page is counted there, once, never here too. A page your firewall refused our bot (a CloudFront “Request blocked”, a Cloudflare challenge) is skipped and said, never counted as failing: that is the AI crawlers get the pages browsers get criterion’s charge, made once. A page listed in the sitemap that errors (a 404 or a 500, whatever it is served as), redirects to nowhere useful, or is served as a download fails. A link from your home page that redirects to a page we already sampled is that page a second time: we pass it over and take the next link, and it never fails. The score is the share of sampled pages that pass, and the detail names each page that didn’t, and why.
The same pages feed the other criteria — readability, structure, structured data, and more — so one sample tells you how the site does beyond its home page. Your two newest posts are read with them whether or not the sample drew them (from your feed, or the newest in your sitemap), at no request more: they take the places of the pages drawn last, so there are still at most ten, and a site that is mostly posts is judged on its posts. If there is no sitemap, we sample up to ten links from the home page instead, and say so. If there is one and we couldn’t read its pages, we say that, with the reason, and never call it your sitemap’s fault. If every address we take from your sitemap fails, this criterion and the sitemap criterion judge those addresses, and the other criteria read up to ten pages your home page links, so they have pages to read.
A one-page site passes on its home page, as long as that returns 200 with its words in the HTML.
Why it matters
A sitemap is a list of pages you’re telling crawlers to visit. When those pages are broken, empty, or the wrong kind of file, a crawler spends its visit on nothing and learns to trust your list less. An AI assistant can only quote what made it into an index, and a page that fails to load never does.
This criterion carries the most weight in Coverage because it is the plainest test of all: you named these pages, so they should work.
How to pass
- Open each page listed under “Which pages”. If it’s gone for good, take it out of the sitemap; if it moved, list the new address instead of the old one.
- Make sure every listed page answers with a 200, not a redirect or an error.
- Serve pages as HTML (or Markdown as
text/markdown), not asapplication/octet-streamor another download type. - Make sure each page has real text in it — at least a few sentences — in the HTML itself, not only after JavaScript runs.
- Generate the sitemap from the pages that exist, so it can’t drift. Most platforms and site generators do this for you.
Evidence
-
Vercel with MERJ · December 2024 · strong
About a third of ChatGPT’s and Claude’s fetches hit 404s, against about 8% of Googlebot’s: the AI crawlers waste much of their visit on pages that aren’t there. Indirect: it measures crawlers, not answers.
One month of server logs across Vercel’s network in 2024. Vercel sells server rendering.
Last changed .