Search and answer bots are allowed
What we check
robots.txt is the file at the root of your site that tells bots where they may go. We read it and ask, for each of the bots that search the web or fetch pages when someone asks an assistant a question, whether it’s allowed at /: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot, and DuckAssistBot.
The score is the fraction of them that are allowed. No robots.txt at all counts as allowing everyone, so it passes. The bots that collect text to train models (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Bytespider, and Amazonbot) are reported for your information, but blocking them costs nothing here.
Why it matters
When someone asks ChatGPT, Claude, or Perplexity about you, the assistant either searches an index built by one of these bots or sends one to fetch your page right then. If robots.txt turns that bot away, it obeys, and the assistant answers without you — or names someone else.
Blocking training is a separate choice, and a reasonable one. The common mistake is a single Disallow: / meant for training bots that catches the search and answer bots too.
How to pass
- Open
/robots.txton your site and look forDisallow: /underUser-agent: *or under any of the bots named above. - Allow the search and answer bots. If you want to keep training bots out, name them separately:
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: CCBot Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml - Check that your platform or a plugin isn’t writing its own robots.txt over yours. Some site builders have a “block AI” switch that blocks more than training.