About our bot

HideAndSeekBot fetches a site when someone asks Hide and Seek for a report on it. It reads what an AI assistant would read, and nothing more.

What it fetches

robots.txt first, always. Then the home page, a sitemap, up to five pages from it, a feed, an about page, llms.txt and ai.txt if there are any, and an IndexNow key file or two. It also checks that http and www lead to the same place, asks for one made-up page to see that it is a 404, asks for the home page as Markdown and looks for a .md copy of a page or two, and fetches the home page once with a browser’s user-agent, to see whether bots are treated differently. Each URL once per scan, and never more than 25 requests to one site.

How often

When someone asks, at most once per site per day: a second request within 24 hours gets the report we already have. Every Monday, for a site whose owner has proved it’s theirs and asked us to watch it. And while an owner is proving a site, the home page every few minutes, for a day, then hourly.

How it identifies itself

HideAndSeekBot/0.1 (+https://hideandseek.bot/bot)

Blocking it

It obeys robots.txt. Add these two lines and the scanner stops after reading robots.txt: nothing else is fetched, and the report says the site asked us not to look.

User-agent: HideAndSeekBot
Disallow: /

A Disallow for a narrower path keeps it out of just that path. A Disallow: / for every bot (User-agent: *) stops it too, unless a group for HideAndSeekBot allows it.

Allowing it

If your CDN or firewall turns bots away, let requests with that user-agent through. Questions go to hello@hideandseek.bot.