Missing pages are 404s
What we check
We ask for one made-up address on your site — a random string of 24 characters that can’t be a real page — and look at the answer. A 404 (“not found”) or 410 (“gone”) passes, and so do other “no” answers like 403, because crawlers treat them as missing too.
If the made-up page answers with a normal 200, it fails: that’s a soft 404, an error page dressed up as a real one. If it redirects to the home page, it scores 0.3. If the server errors or times out, or robots.txt asks us not to fetch that address, we skip the check.
Why it matters
Crawlers trust the status code more than the words on the page. When a missing page says “200 OK”, a crawler files your “Page not found” message as real content, sometimes thousands of times over, and spends its visits on nothing instead of on the pages that matter.
Redirecting every missing page to the home page has the same effect, and it also hides broken links you’d want to know about.
How to pass
- Make your site’s “not found” page return a real 404 status. The page can still be friendly and helpful; only the status code needs to change.
- Don’t redirect unknown addresses to the home page. Redirect only pages that moved, each to its new address.
- To check, ask for a made-up address and look at the status:
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/no-such-page-here
It should print 404.