Search and answer bots are allowed

Does robots.txt let in the bots that search and answer questions?

Access · 30 of 100 in Access · yours to do · robots

What we look for

robots.txt is the file at the root of your site that tells bots where they may go. We read it and ask, for each of the bots that search the web or fetch pages when someone asks an assistant a question, whether it’s allowed at / and at each of the other pages we read (the sampled pages and your feed’s newest posts, each tested against its own host’s robots.txt, read once per host, so a help center on another subdomain is judged by its own file; a host whose file we couldn’t read is left out and said), each with its query string, as the robots.txt standard matches them): OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Agent, Bingbot, DuckAssistBot, Applebot, Amzn-SearchBot, Amzn-User, Meta-WebIndexer, Meta-ExternalFetcher, MistralAI-Index, and MistralAI-User, each taken from its operator’s own documentation. If your home page moves to another host (example.com to www.example.com), we read that host’s robots.txt, since it’s the one that governs your pages. We also say how many AI bots the file names, and how many it turns away.

The score is by assistant, not by agent: ChatGPT, Claude, Perplexity, Google, and Microsoft count double, DuckDuckGo, Apple, Amazon, Meta, and Mistral once, and an assistant’s share is the fraction of its agents allowed, so a site that blocks only Applebot loses a little, and letting in the agents a site allows by default never offsets a block on the assistants people ask most. An agent its operator says may ignore robots.txt when a user asks (Amzn-User, Meta-ExternalFetcher) is still reported when you block it: you asked it not to come. No robots.txt at all (a 404) counts as allowing everyone, so it passes; so does a robots file that your server answers with a 404, and we say so, since crawlers won’t read it. A robots.txt we can’t read — a firewall’s 403 page, or an HTML page at its address — isn’t a robots.txt that allows everyone, so the criterion is left unscored and says so. The bots that collect text to train models (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, MistralAI-Training, Bytespider, and Amazonbot) are reported for your information, but blocking them costs nothing here.

Access is scored by who you let in. Every other Access criterion’s points are earned in proportion to this one’s score: Access is this criterion’s points, plus that share of the points the others earned. A firewall that lets a bot through is worth to an assistant only what your robots.txt lets that assistant use. A worked example: a site that turns away ChatGPT, Claude, Perplexity, and four of the smaller assistants keeps 5 of 15 by assistant, a third. It scores 10 of 30 here and keeps a third of the 70 the other Access criteria can earn, so its Access is at most 33 of 100, where it could have read 80. A site that turns every assistant away has an Access of 0; one that turns nobody away loses nothing; and when this criterion is left unscored, nothing is scaled. Who reads your site is your choice to make, and the report says it that way.

Why it matters

The AI companies document two kinds of bot: the ones that answer and the ones that train. OpenAI, Anthropic, Perplexity, and Google each say which of theirs puts your pages in answers and which only collects text for training (the evidence is below). Few tools make the split; this one does.

When someone asks ChatGPT, Claude, or Perplexity about you, the assistant either searches an index built by one of these bots or sends one to fetch your page right then. If robots.txt turns that bot away, it obeys, and the assistant answers without you — or names someone else.

Blocking training is a separate choice, and a reasonable one. The common mistake is a single Disallow: / meant for training bots that catches the search and answer bots too.

How to pass

  1. Open /robots.txt on your site and look for Disallow: / under User-agent: * or under any of the bots named above.
  2. Allow the search and answer bots. If you want to keep training bots out, name them separately:
    User-agent: GPTBot
    User-agent: ClaudeBot
    User-agent: Google-Extended
    User-agent: CCBot
    Disallow: /
    
    User-agent: *
    Allow: /
    
    Sitemap: https://example.com/sitemap.xml
  3. Check that your platform or a plugin isn’t writing its own robots.txt over yours. Some site builders have a “block AI” switch that blocks more than training.

Evidence

  • OpenAI, crawler documentation · 2026 · strong

    OAI-SearchBot is what puts sites in ChatGPT’s search results; GPTBot collects training data only; ChatGPT-User fetches a page when a user asks.

    Vendor documentation.

  • Anthropic, crawler documentation · February 2026 · strong

    ClaudeBot collects training data; blocking Claude-SearchBot or Claude-User may make a site less visible in Claude’s answers.

    Vendor documentation.

  • Perplexity, crawler documentation · 2026 · strong

    PerplexityBot is what surfaces sites in Perplexity’s answers, and it isn’t used to train models.

    Vendor documentation.

  • Google, “AI features and your website” · December 2025 · strong

    Google-Extended governs training only: blocking it doesn’t affect Search or AI Overviews.

    Vendor documentation.

  • Apple, “About Applebot” · September 2026 · strong

    Applebot indexes for Siri, Spotlight, and Safari; Applebot-Extended is the token that opts out of training Apple’s models.

    Vendor documentation.

  • Amazon, “Amazonbot” · October 2026 · strong

    Amzn-SearchBot indexes for Alexa and isn’t used for training; Amzn-User fetches a page when a user asks and may not follow every directive; Amazonbot may be used to train Amazon’s models.

    Vendor documentation.

  • Meta, “Meta web crawlers” · 2026 · strong

    Meta-WebIndexer indexes for Meta AI search; Meta-ExternalFetcher fetches a link at a user’s request and may bypass robots.txt; meta-externalagent trains and indexes.

    Vendor documentation.

  • Mistral, “Robots” · 2026 · strong

    MistralAI-Index indexes for search and MistralAI-User fetches at a user’s request, neither for training; MistralAI-Training trains.

    Vendor documentation.

Last changed .

The whole rubric