Robots.txt Checker: Find blocking errors.
Free robots.txt validator. We parse your robots.txt, validate its syntax, and cross-check every Disallow rule against your sitemap to find URLs you are accidentally blocking.
What a robots.txt audit really tells you
Robots.txt is two lines of text that can quietly destroy your search traffic if they’re wrong. The most common SEO disasters we see — easily six figures of lost monthly traffic — come from a single mis-typed Disallow rule, a stale staging directive that never got removed on production launch, or a robots.txt that blocks a CSS/JS folder Google now needs to render the page properly.
Our checker fetches the live robots.txt, parses it (catching syntax errors that Google would silently ignore), and cross-checks every Disallow rule against your actual sitemap URLs. If your sitemap lists 1,200 URLs and your robots.txt would block 80 of them, we tell you exactly which 80 and why — usually a trailing-wildcard rule that’s broader than intended.
Common robots.txt mistakes we catch
- Disallow: / — the nuclear option, blocks the entire site. Usually a leftover from staging.
- Pattern over-reach —
Disallow: /admin*matches/administrative-servicestoo. - Case sensitivity issues — paths are case-sensitive;
Disallow: /Blogwon’t block/blog. - Sitemap declaration mismatches —
Sitemap:URL doesn’t match what’s actually live. - AI bot directives — accidentally blocking GPTBot/ClaudeBot/Google-Extended cuts AI search visibility while keeping traditional Google indexing intact.
- Blocked CSS/JS — Googlebot needs both to render modern pages; blocking them tanks Core Web Vitals scoring.
Why we cross-check against your sitemap
A robots.txt that’s valid in isolation can still be self-defeating: the moment a sitemap-listed URL gets blocked by a Disallow rule, Google sees a contradiction. The sitemap says “please index this” and robots.txt says “please don’t crawl this”. Google will resolve the conflict by neither crawling nor indexing — the exact opposite of what either signal intended. We catch every conflict and rank them by how many URLs are affected.