Blocked Due to Access Forbidden (403) or Unauthorized Request (401): How to Fix It
Short answer: your server refused Googlebot. With a 403 it said “access forbidden”; with a 401 it asked for a login. Googlebot never logs in, so either way it can’t read the page, and Google won’t index it. If the page should be public, find what blocks Googlebot (a firewall, bot protection, password protection or a login rule) and let it through. If the page is meant to be private, the status is correct.
What the two statuses mean
Both show up in the Page indexing report (Indexing → Pages) as reasons a page is not indexed. They are two versions of the same problem, so this page covers both.
Blocked due to access forbidden (403)
A 403 normally means a visitor sent a login but still has no access. Google’s help points out the catch: “Googlebot never provides credentials, so your server is returning this error incorrectly.” In other words, something on your side is refusing Googlebot on purpose, most often a firewall or bot rule.
Blocked due to unauthorized request (401)
Google’s help says: “The page was blocked to Googlebot by a request for authorization.” The server asked for a username and password, or a login token. Common sources are password protection on a host, a staging login that went live by mistake, or a members-only area.
Google’s HTTP status docs explain what happens next. Google does not index URLs that return a 4xx code, removes them if they were indexed, and slowly crawls them less often. All 4xx codes except 429 are handled the same way.
Step 1: is the block on purpose?
Look at the example URLs in the report. Login pages, account areas, admin panels and staging copies should block Google. For those, the fix is to remove them from your sitemap and internal links, not to open them up. Public pages such as articles, products and your home page must not be here.
Step 2: reproduce what Googlebot gets
- Open the URL in a private window. Google’s help suggests this for 401s. A login box or password page means the page is gated for everyone who is not logged in.
- Run URL Inspection → Test live URL in Search Console. It fetches the page as Googlebot from Google’s servers and shows the status code it got.
- Check the response headers. A 401 usually comes with a
WWW-Authenticateheader. A 403 from a firewall or CDN often has the provider’s name in the headers or the error page.
curl -sI https://example.com/page # HTTP/2 401 # www-authenticate: Basic realm="Staging" <- password protection is on
A plain curl from your computer is not Googlebot. If curl gets 200 but the live test gets 403, the block is aimed at Google’s IPs or user agent, which points to a firewall or bot rule.
Step 3: fix the cause
Firewall, WAF or bot protection (usually 403)
Bot-fighting features, rate limits, country blocks and IP allowlists are the most common reason Googlebot gets a 403. Check the firewall log for blocked requests from Googlebot and add an exception for verified Googlebot. Verify it the way Google describes: a reverse DNS lookup on the IP should return a host in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that host should return the same IP. Google also publishes its crawler IPs in common-crawlers.json. Don’t allow anything that just claims to be Googlebot in its user agent.
Password protection left on (usually 401)
Hosts and frameworks make it easy to password-protect a preview or staging site. If that protection stays on for the live domain, every visitor without a password, Googlebot included, gets a 401. Turn it off for the production domain, or remove the auth_basic (nginx) or AuthType (Apache) rules that cover public pages.
Login rules that cover too much
Middleware or a plugin that requires a session can catch public routes by mistake. Check which paths the rule matches and exclude the public ones.
Members-only content you want indexed
Google can only index what it can fetch. To have paywalled pages indexed, serve the full content to verified Googlebot and mark the locked part with structured data: isAccessibleForFree set to false, plus hasPart with a cssSelector for the paywalled section. Google says this markup helps it tell a paywall apart from cloaking.
One more rule from Google’s docs: don’t use 401 or 403 to slow Google’s crawling. It won’t slow it, and it drops the pages from the index.
Step 4: confirm and validate
Run the live test again. When it returns 200, click Request indexing for key URLs and Validate fix in the report. Google says validation usually takes up to about two weeks.
To catch this before Google does, the free sitemap finder requests 30 URLs from your sitemap and lists any that answer with 401, 403 or another error. It requests them from our server, not from Google’s IPs, so a block aimed only at Googlebot can still pass our test. Use the live test in Search Console for that case.
Frequently asked questions
What is the difference between the 403 and the 401 status?
A 401 asks for a login (the server sends an authorization request). A 403 refuses access outright. For Google the result is the same: Googlebot never logs in, so the page is not crawled or indexed.
The page opens fine for me. Why does Google get a 403?
Your browser may be logged in, or your IP and country may be allowed while Googlebot’s are not. Firewalls and bot protection often block crawlers on purpose. Test in a private window and check your firewall logs for Googlebot requests.
Can I use 403 or 401 to slow Googlebot down?
No. Google says not to use 401 or 403 to limit crawl rate. These codes make Google treat the URL as unavailable and drop it from the index.
How do I get members-only content indexed?
Google has to be able to fetch the full content. Serve it to verified Googlebot and mark the paywalled part with structured data (isAccessibleForFree set to false, plus hasPart with a cssSelector). That markup helps Google tell a paywall apart from cloaking.
Should private pages return 403 or 401?
Yes, if they must stay private. Google does not index URLs that return a 4xx code. Also remove them from your sitemap and internal links, so they stop showing in this report.