AI crawlers get the pages browsers get
Do AI’s crawlers get the same sampled pages a browser gets?
What we look for
The edge criterion asks this of your home page; this one asks it of two of the pages we sample, the first two, in requests the scan keeps for it. We fetch each twice more: once as a plain bot, the way an AI crawler arrives, and once as an ordinary Chrome browser, each once.
The bot gets what the browser gets when the two answers are of the same kind (both a page, both a redirect, both not found) and, for a page, the same document: both with or both without a <main>, and with visible text within a fifth of each other. A bot that gets the page where the browser is refused or challenged is no bot problem: that page passes. The score is the share of the two pages where they agree, and the sentence names the pages: “/2026/11/a and /about answer our bot the same as a browser.”
A firewall refusing the bot is charged here, once (and in edge for the home page): the pages and sitemap criteria skip an address a firewall refused, never counting it missing or failing. If the scan’s request budget runs out first, we judge the page we compared and say so.
Why it matters
A firewall, a CDN rule, or a prerender service can treat bots differently from people without anyone noticing: the home page is fine, and the pages deeper in the site answer a crawler with an error, a challenge, or a different page. AI crawlers hit errors far more often than search crawlers do. Every page that answers them badly is a page an assistant can’t read or cite.
How to pass
- Open each page the report names in a browser, then fetch it the way a bot does:
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; ExampleBot/1.0)" https://example.com/pricing - If the bot gets an error or a challenge, look at what’s in front of the site: a CDN or firewall rule, a bot-management setting, or rate limiting that applies only to bots. Allow the search and user agents you want.
- If the bot gets a different page, look at any prerender or “bot version” service: it should serve bots the page a browser gets.
Evidence
-
Vercel with MERJ · December 2024 · strong
AI crawlers hit errors about four times as often as Googlebot: about 34% of ChatGPT’s and Claude’s fetches were 404s, against about 8% of Googlebot’s. Indirect: it measures crawlers, not answers.
One month of server logs across Vercel’s network in 2024. Vercel sells server rendering.
Last changed .