Your firewall lets bots in

Does the home page reach a bot the same way it reaches a browser?

Access · 20 of 100 in Access · yours to do · edge

What we look for

We fetch your home page twice: once as our own bot, HideAndSeekBot, and once as an ordinary Chrome browser. If the browser gets the page and the bot gets turned away — an error like 403, 429, or 503, a “Just a moment” or “Checking your browser” challenge, an “Access denied” page, or a near-empty page where the browser got real text — the criterion fails. If both get in, it passes. Both fetches speak HTTP/2, as browsers and AI fetchers do, and the HTTP/2 answer is the one judged: a home page that challenges HTTP/2 and lets an older client through (“Browsers and AI fetchers are challenged at https://example.com/ (HTTP/2 answers 403); an older client gets through.”) is turned away here, whatever HTTP/1.1 would get. If both are turned away, and every other request with them, nothing is scored: the report says in one sentence that every request was refused, and gives no score until the site answers.

We only ever test as ourselves. We never pretend to be another company’s bot. And for this one fetch we leave off the signature our bot usually carries (Web Bot Auth, which lets a firewall verify it), because the question is what an unverified bot gets, which is what most AI crawlers are. So we can’t see a server that refuses AI crawlers by their names (GPTBot, ClaudeBot, PerplexityBot): only those crawlers would meet it.

When both the bot and the browser get the page, this criterion passes, even where the two documents differ (a <main> in one, more text in the other): that is the AI crawlers get the pages browsers get criterion’s question, asked of the sampled pages, and this one never fails a site whose bot copy is the better one.

Why it matters

robots.txt is the polite front door, but many sites have a second one in front of it: a CDN or firewall, like Cloudflare, that blocks traffic it thinks is automated. It decides before robots.txt is ever read. Your robots.txt can welcome every assistant while your firewall quietly shows them a challenge page.

An assistant that gets a challenge page has nothing to read. It doesn’t retry, and it doesn’t tell anyone. You just don’t come up.

How to pass

  1. Find out what sits in front of your site: Cloudflare, Fastly, Akamai, Vercel, a WordPress security plugin, or your host’s firewall.
  2. On Cloudflare, go to Security → Bots and allow AI crawlers, or add a rule that skips bot protection for the agents you want.
  3. On other firewalls, allow-list the search and answer bots by user-agent: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Agent, Bingbot, DuckAssistBot, Applebot, Amzn-SearchBot, Amzn-User, Meta-WebIndexer, Meta-ExternalFetcher, MistralAI-Index, and MistralAI-User.
  4. Scan again. A rule that only allows “verified bots” may still turn away the newer assistants.

Evidence

Last changed .

The whole rubric