Your firewall lets bots in
What we check
We fetch your home page twice: once as our own bot, HideAndSeekBot, and once as an ordinary Chrome browser. If the browser gets the page and the bot gets turned away — an error like 403, 429, or 503, a “Just a moment” or “Checking your browser” challenge, an “Access denied” page, or a near-empty page where the browser got real text — the check fails. If both get in, it passes. If both are turned away, we skip it, because the site isn’t answering anyone.
We only ever test as ourselves. We never pretend to be another company’s bot.
Why it matters
robots.txt is the polite front door, but many sites have a second one in front of it: a CDN or firewall, like Cloudflare, that blocks traffic it thinks is automated. It decides before robots.txt is ever read. Your robots.txt can welcome every assistant while your firewall quietly shows them a challenge page.
An assistant that gets a challenge page has nothing to read. It doesn’t retry, and it doesn’t tell anyone. You just don’t come up.
How to pass
- Find out what sits in front of your site: Cloudflare, Fastly, Akamai, Vercel, a WordPress security plugin, or your host’s firewall.
- On Cloudflare, go to Security → Bots and allow AI crawlers, or add a rule that skips bot protection for the agents you want.
- On other firewalls, allow-list the search and answer bots by user-agent: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, and DuckAssistBot.
- Scan again. A rule that only allows “verified bots” may still turn away the newer assistants.