Blocked by robots.txt in Google Search Console: How to Fix It
Short answer: a rule in your robots.txt file tells Googlebot not to crawl this URL, so Google did not fetch the page. That is fine for pages you don’t want crawled, like admin or cart pages. If the page should be in search, find the Disallow rule that matches it, remove or narrow that rule, and ask Google to recrawl. If you want the page out of search, robots.txt is the wrong tool: use noindex instead.
What the status means
Search Console lists it in the Page indexing report (Indexing → Pages) under the reasons pages are not indexed. Google’s help says: “This page was blocked by your site’s robots.txt file.”
Before Googlebot crawls a URL, it reads /robots.txt on that host. If a Disallow rule matches the path, it stops there. Google never sees the page content, so it can’t rank it for anything.
One catch: a block does not promise the URL stays out of the index. Google’s help notes that the page could still be indexed “through some other means”, such as links from other sites. When that happens, the URL moves to a separate status, Indexed, though blocked by robots.txt.
Step 1: check if the block is on purpose
Open the status and look at the example URLs. Sort them into two groups:
- Blocked on purpose: login, admin, cart, checkout, internal search results, filter and sort URLs. Leave these alone.
- Blocked by mistake: blog posts, product pages, category pages, or anything listed in your sitemap. Fix these.
A quick sign of a mistake: the blocked URL is also in your XML sitemap. A sitemap says “please crawl this”, robots.txt says “don’t”. The free robots.txt checker tests 30 URLs from your sitemap against the rules in your User-agent: * group and lists the ones that are blocked.
Step 2: find the rule that blocks the page
Run URL Inspection on one affected URL. Under Page indexing, “Crawl allowed?” will say No: blocked by robots.txt. That confirms the block, but it does not show which line causes it. To find the line, open https://yourdomain.com/robots.txt and read it with these rules in mind:
Googlebot follows one group only
If the file has a User-agent: Googlebot group, Googlebot uses only that group and ignores User-agent: *. So a page can be allowed under * and still be blocked for Googlebot.
The most specific rule wins
When an Allow and a Disallow rule both match, Google uses the longer (more specific) one. If they are equally long, Allow wins.
Rules match the start of the path
Disallow: /p blocks /products/, /pricing and /p itself. Google also supports * (any characters) and $ (end of URL). A rule like Disallow: /*? blocks every URL with a question mark.
# Risky rules - each one blocks more than it looks like User-agent: * Disallow: /p # also blocks /products/ and /pricing Disallow: /*? # blocks every URL with a query string Disallow: / # blocks the whole site (common staging leftover) # Narrow rules - block only what you mean User-agent: * Disallow: /admin/ Disallow: /cart Allow: /admin/help/ # longer rule wins, so this folder stays crawlable
Google’s own fix guide also suggests a robots.txt validator: enter the page URL and it shows which rule blocks it.
Step 3: change the rule
How you edit robots.txt depends on your setup:
- You can edit the file: remove the rule, or narrow it. For example, change
Disallow: /blog/toDisallow: /blog/drafts/if only drafts should be hidden. - Hosted platforms (Wix, Shopify, Squarespace and others) manage robots.txt for you. Check your platform’s docs for what you can change.
- WordPress: look for rules added by an SEO plugin or a physical
robots.txtfile in your site root. The “Discourage search engines” setting no longer writes to robots.txt (it adds a noindex tag since WordPress 5.3), so it is not the cause here. See our WordPress robots.txt guide.
Want the page hidden from search instead? Remove the robots.txt block and add <meta name="robots" content="noindex"> to the page. Google must be able to crawl the page to see a noindex tag. Our noindex tag guide explains both options.
Step 4: get Google to see the change
Google generally caches robots.txt for up to 24 hours. To speed things up, open Settings → robots.txt in Search Console. This report shows the robots.txt files Google found for the top 20 hosts of your property, when each was last checked, and any parsing problems. From a file’s menu you can choose Request a recrawl. Google says it doesn’t guarantee an instant refetch. The report is only available for domain-level properties.
Then inspect the URL again and run a live test. If “Crawl allowed?” now says Yes, click Request indexing. You can also click Validate fix in the report. Google says validation usually takes up to about two weeks.
Frequently asked questions
Can a page blocked by robots.txt still be indexed?
Yes. Google can index the URL without crawling it, if other pages link to it. The result then shows no description. Search Console reports that case as "Indexed, though blocked by robots.txt".
Should I use robots.txt or noindex to keep a page out of Google?
Use noindex. Google says robots.txt is not a way to keep a page out of Google. Noindex only works if Google can crawl the page, so the page must not be blocked in robots.txt at the same time.
How fast does Google see my robots.txt change?
Google generally caches robots.txt for up to 24 hours. You can ask for a faster refetch in the robots.txt report in Search Console, but Google does not promise an instant recrawl.
Does "Discourage search engines" in WordPress block pages in robots.txt?
Not since WordPress 5.3. The setting now adds a noindex robots meta tag to your pages instead of a Disallow rule. If you see Blocked by robots.txt on WordPress, look for a rule added by a plugin, a physical robots.txt file, or your host.
Is it bad to have URLs under this status?
Only if they are pages you want in search. Blocks on admin, cart, internal search or filter URLs are often on purpose and fine to leave.