* Robots.txt Checker

Robots.txt Checker: Find blocking errors.

Free robots.txt checker. We read your robots.txt and test 30 URLs from your sitemap against its Disallow rules, so you see pages you are accidentally blocking.

Try:
✓ No signup for your first check✓ Full sitemap + 30 live URL checks✓ No credit card

What a robots.txt audit really tells you

Robots.txt is a few lines of text that can quietly destroy your search traffic if they’re wrong. Some of the costliest SEO mistakes come from a single mistyped Disallow rule, a staging directive that never got removed at launch, or a rule that blocks a CSS/JS folder Google needs to render the page properly.

Our checker fetches your live robots.txt, reads the Disallow rules in the User-agent: * group, and tests 30 URLs sampled evenly from your sitemap against them. Any sitemap URL that robots.txt blocks is flagged, because the two files then contradict each other. It matches rules as plain path prefixes: wildcard rules (*, $) and groups for named crawlers such as Googlebot or GPTBot are not evaluated, and it doesn’t check syntax.

Common robots.txt mistakes to look for

  • Disallow: / — the nuclear option, blocks the entire site. Usually a leftover from staging.
  • Pattern over-reach — Disallow: /admin also matches /administrative-services, because rules match path prefixes.
  • Case sensitivity issues — paths are case-sensitive; Disallow: /Blog won’t block /blog.
  • Sitemap declaration mismatches — the Sitemap: URL doesn’t match what’s actually live.
  • AI bot directives — blocking GPTBot, ClaudeBot or Google-Extended changes what AI products can use, even though traditional Google indexing is unaffected. Decide on purpose.
  • Blocked CSS/JS — Googlebot needs both to render modern pages; blocking them can hide content from Google.

Why we cross-check against your sitemap

A robots.txt that’s valid in isolation can still be self-defeating: the moment a sitemap-listed URL gets blocked by a Disallow rule, Google gets two opposite signals. The sitemap says “please index this” and robots.txt says “don’t crawl this”. Google won’t crawl the page, but it can still index the bare URL if links point to it — the “Indexed, though blocked by robots.txt” status in Search Console. We flag each conflict we find in the sample.

Common questions

What does the robots.txt checker analyze?
We fetch your robots.txt, read the Disallow rules in the User-agent: * group, and test 30 URLs sampled from your sitemap against them. A sitemap URL that robots.txt blocks is flagged.
Can I check robots.txt without a sitemap?
No. The check needs a sitemap to test URLs against; without one the report stops at "no sitemap found". You can still read the file yourself at yourdomain.com/robots.txt.
Will it detect conflicts with my sitemap?
Yes, for the 30 sampled URLs. Rules are matched as plain path prefixes; wildcard rules (* and $) and groups for named crawlers such as Googlebot or GPTBot are not evaluated yet.