“Sitemap Contains URLs Which Are Blocked by robots.txt”: How to Fix It

Short answer: your XML sitemap lists URLs that your robots.txt tells Google not to crawl. The two files contradict each other, and robots.txt wins, so those URLs don’t get crawled. Find which URLs are affected and which rule blocks them. Then decide per URL: if the page should be in Google, remove or narrow the robots.txt rule; if it shouldn’t, remove it from the sitemap.

Where you see this warning

Google’s Sitemaps report help lists it word for word: “Sitemap contains urls which are blocked by robots.txt”, meaning Google couldn’t access all the content in your sitemap because robots.txt blocks it. You find it in Search Console under Indexing → Sitemaps, by opening the sitemap’s details.

The same problem also shows up URL by URL. In the Page indexing report, set the sitemap filter at the top to All submitted pages (or pick your sitemap). Any URL listed under “Blocked by robots.txt” is in your sitemap and blocked at the same time. The old report called this “Submitted URL blocked by robots.txt”; that name no longer exists.

Two different problems with similar names

1. URLs inside the sitemap are blocked

This is the warning above. The sitemap loads fine, but some of the URLs it lists match a Disallow rule. Fix it with the steps below.

2. The sitemap file itself is blocked

If a rule matches the sitemap’s own path, for example Disallow: /sitemap or Disallow: /*.xml$, Google can’t read the file at all. Google’s help names a robots.txt block as one cause of the “Couldn’t fetch” status. Remove that rule, or add an Allow line for the sitemap path. See Couldn’t fetch and could not be read.

Step 1: find the blocked URLs and the rule

1. In the Page indexing report, filter to All submitted pages and open “Blocked by robots.txt”. Export the examples.
2. Open https://yourdomain.com/robots.txt and find the Disallow rule that matches each URL path. Remember that Googlebot follows only the most specific user-agent group, and that rules match from the start of the path.
3. Look for a pattern. Usually one rule blocks a whole section: /tag/, /search, /*? or a staging leftover like Disallow: /.

The free robots.txt checker does the comparison for you on 30 URLs from your sitemap: it reads your live robots.txt and lists sitemap URLs that the User-agent: * rules block.

Step 2: decide for each group of URLs

The pages should be in Google

The robots.txt rule is too broad. Remove it, or narrow it so it only covers what you meant to block:

# Before: blocks every URL with a query string, including real pages
Disallow: /*?

# After: block only the sort and filter parameters
Disallow: /*?sort=
Disallow: /*?filter=

The pages should not be in Google

Then the block may be right, but the sitemap is wrong. Remove these URLs from the sitemap so it lists only pages you want indexed. Common culprits are tag and search pages, cart and account pages, and parameter URLs that a plugin adds to the sitemap. If you want these pages kept out of the index for good, robots.txt is the wrong tool anyway: use noindex, and let Google crawl the page to see it. See Indexed, though blocked by robots.txt.

Step 3: tell Google and confirm

1. If you changed robots.txt, open Settings → robots.txt in Search Console and use Request a recrawl on the file.
2. If you changed the sitemap, resubmit it in the Sitemaps report.
3. Run URL Inspection with a live test on one fixed URL. “Crawl allowed?” should now say Yes.

Fix the cause at the source, not just in the file. If a CMS plugin keeps adding blocked URL types to the sitemap, turn that content type off in the plugin’s sitemap settings, or the warning comes back on the next rebuild.

Frequently asked questions

Related guides

• Blocked by robots.txt (Page indexing status)
• Indexed, Though Blocked by robots.txt
• How to Add Your Sitemap to robots.txt
• Sitemap Could Not Be Read or Couldn’t Fetch
• All sitemap errors