In Common Crawl
What we check
Common Crawl is a free, public copy of a large part of the web, refreshed every month or two. We ask its index whether the latest crawl has any pages from your domain, including its subdomains. If it does, the check passes and we tell you roughly how many pages it holds (up to “1,000+”); if not, it fails. If the index is down or slow to answer, we skip the check.
Why it matters
Common Crawl is one of the main sources of text that AI models are trained on. A site that isn’t in it is, for many models, a site that was never read. What a model knows about you before it searches anything comes, in large part, from crawls like this one.
It’s also a fair signal of how visible a site is to crawlers in general: Common Crawl follows links, so a site nothing links to tends to be missing.
How to pass
There’s no direct fix. You can’t submit a site to Common Crawl; it has to exist long enough and be linked from enough other places to be found. What helps:
- Don’t block it. Common Crawl’s crawler is
CCBot; if your robots.txt blocks it, you won’t be in the archive. Blocking it is a fair choice if you don’t want to be in training data, but know the trade.User-agent: CCBot Allow: / - Get linked. Everything else in Authority — a Wikidata item, directory listings, profiles that link back — gives crawlers more ways in.
- Keep the basics right: a sitemap, pages that load, and no firewall turning bots away. Then give it a crawl or two.