Indexed, Though Blocked by robots.txt: What It Means and How to Fix It
Short answer: Google put this URL in its index even though your robots.txt tells it not to crawl the page. It found the URL through links, so it indexed the address without reading the content. The fix depends on what you want. If the page should rank, remove the robots.txt block. If it should not be in Google at all, remove the block and add noindex, because Google says “robots.txt is not the correct mechanism to avoid being indexed.”
What the status means
Search Console shows this as a warning in the Page indexing report, not as a “not indexed” reason. The page is in Google’s index. Google’s help describes it this way: the page “was indexed despite being blocked by your website’s robots.txt file.”
Robots.txt only controls crawling (fetching the page). Indexing is a separate step. Google can index a URL it never fetched, using what it learns from links to that URL. Because Google can’t read the page, the search result shows no description. Google’s robots.txt guide says such a result “won’t have a description.”
Compare it with Blocked by robots.txt: same block, but there the URL is not indexed.
Why it happens
- Links to a blocked URL. Other sites, or your own pages, link to a URL your robots.txt blocks. Old URLs that still have backlinks are a common case.
- A rule that is too broad. A rule like
Disallow: /*?orDisallow: /pblocks real pages you meant to keep open. - Blocking pages to hide them. Login, search or staging URLs were blocked in robots.txt to keep them out of Google. That only stops crawling, so linked URLs can still show up.
- Blocked URLs in your sitemap. Listing a blocked URL in the sitemap tells Google the URL exists and matters, while robots.txt blocks it.
Fix 1: the page should be in search
Then the block is a mistake. Find the Disallow rule that matches the URL and remove or narrow it. Our Blocked by robots.txt guide shows how Google picks the rule that applies. After the change, inspect the URL in Search Console, run a live test, and click Request indexing. Once Google can crawl the page, it gets a normal title and description.
Fix 2: the page should not be in search
This is the one most people get wrong. The order matters:
- Remove the robots.txt block for the URL, so Googlebot can fetch it.
- Add noindex to the page, or send it as a header. Either line below works:
<meta name="robots" content="noindex"> X-Robots-Tag: noindex
- Check with URL Inspection and a live test that crawling is allowed and noindex is detected.
- Keep the page crawlable. If you block it again, Google can’t see the noindex any more, and links can put the URL back in the index.
Adding noindex while the robots.txt block is still there does nothing. Google never fetches the page, so it never sees the tag. Our noindex tag guide and X-Robots-Tag guide cover the setup.
Private pages need more than noindex
Anyone can read your robots.txt, so it can point people to the URLs you want hidden. For private content, use a password or login. Google’s robots.txt guide names password protection as the other way to keep a page out of Google. A login page that answers with a 401 or 403 is not indexed: Google’s HTTP status docs say it does not index URLs that return a 4xx code.
Need it gone right now?
The Removals tool in Search Console hides a URL fast, but only for about six months. Use it as a stopgap while noindex, a password, or a 404/410 does the permanent job.
The old “Noindex:” line in robots.txt
Some old guides tell you to write Noindex: /path/ inside robots.txt. Google stopped supporting that on September 1, 2019. If your file still has these lines, they do nothing. Move the rule to a meta robots tag or an X-Robots-Tag header.
When you can leave it
If the URLs are not worth a click, such as old redirect or asset URLs, and you don’t mind them showing up without a description, you can leave the warning. It is not an error. Fix it when the URLs are pages people could find and click, or when they should be private.
How to find the affected URLs
In Search Console, open Indexing → Pages, click the warning, and export the list. To check where the URLs come from, compare them with your sitemap. The free robots.txt checker tests 30 URLs from your sitemap against your User-agent: * rules and lists the blocked ones. A blocked URL in your sitemap is the clearest mistake to fix first.
Frequently asked questions
Why is my page indexed if robots.txt blocks it?
Robots.txt controls crawling, not indexing. Google can add a URL to its index without fetching it, based on links that point to it. The search result then shows no description.
How do I remove a page that is indexed though blocked?
Remove the robots.txt rule that blocks it, then add a noindex meta tag or X-Robots-Tag header. Google has to crawl the page to see the noindex. Once it does, it drops the page from the index.
Can I block the page in robots.txt again after it drops out?
It is risky. Once the page is blocked again, Google can no longer see the noindex, and links can bring the URL back as indexed but blocked. Keep the page crawlable with noindex.
Is this the same as "Blocked by robots.txt"?
No. Both mean robots.txt blocks crawling. "Blocked by robots.txt" pages are not indexed. "Indexed, though blocked" pages are in the index anyway, with almost no information.
Does the Removals tool fix it?
Only for a while. Google says a temporary removal lasts about six months. To keep the page out for good, use noindex, password protection, or a 404/410 status.