Crawl Errors: Every Type Explained and Fixed
Where to Find Crawl Errors in Google Search Console
Crawl errors in Google Search Console are found under Settings → Crawl Stats → By Response. This report shows a breakdown of all Googlebot requests to your site over the past 90 days, grouped by HTTP response code. Click any response code category to see the specific URLs that returned that status during crawling.
Each response category tells a different story about your site's health:
- 2xx (Success) — pages Googlebot successfully accessed. The goal is to have the overwhelming majority of crawl requests here.
- 3xx (Redirects) — URLs that redirect elsewhere. A high redirect rate wastes crawl budget and may indicate internal links pointing to old URLs.
- 4xx (Client errors) — URLs returning not-found or forbidden. These waste crawl budget and should be minimized.
- 5xx (Server errors) — your server failed to respond correctly. These are urgent — repeated 5xx errors cause Googlebot to slow crawl rate across your entire site.
Fix in priority order: 404s appearing in your sitemap first, then 5xx server errors, then 403 access errors, then high redirect counts from internal links.
Cause: The URL does not exist on your server.
Fix: If the page was deleted: create a 301 redirect to the most relevant existing page. If the URL was changed: redirect old URL to new. If it is linked from another site: set up a redirect to capture that link equity. Remove 404 URLs from your sitemap immediately. Do not leave important pages returning 404 - you lose all backlink value.
Cause: Server is blocking Googlebot from accessing the URL due to access restrictions.
Fix: Check your server configuration (.htaccess, nginx config) for IP blocks or authentication requirements. Ensure Googlebot's IP ranges are not blocked. If the page requires authentication, add a noindex tag and remove from your sitemap - authenticated content should not be indexed.
Cause: Your server crashed or errored when Googlebot requested the page.
Fix: Check your server error logs for the specific error during the crawl time. Common causes: database connection failures, PHP memory limits exceeded, plugin conflicts in WordPress, or timeouts on slow database queries. Fix the underlying server issue, then use URL Inspection to request re-crawl.
Cause: Google cannot resolve your domain name - your DNS is unreachable or misconfigured.
Fix: Check your DNS settings at your registrar. Verify nameservers are correctly pointing to your hosting provider. DNS propagation can take up to 48 hours after changes. Use tools like whatsmydns.net to check DNS propagation across global locations.
Cause: Your server is too slow to respond within Google's crawl timeout window.
Fix: Improve server response time (TTFB). Enable server-side caching, upgrade hosting if chronically overloaded, use a CDN. Check if timeouts correlate with high traffic periods - you may need auto-scaling or better infrastructure.
Cause: Google cannot retrieve your robots.txt file.
Fix: Verify your robots.txt is accessible at yoursite.com/robots.txt. Check it returns a 200 status (not 404 or 500). Ensure your server is not blocking requests to /robots.txt. If robots.txt is unavailable, Google defaults to allowing all crawling.
How to Monitor Crawl Errors Proactively
In Google Search Console under Settings, then Crawl Stats, check the Crawl Requests Breakdown by Response table weekly. A healthy site should have the vast majority of responses as 200 (success). More than 5-10% of crawl requests returning errors signals a structural problem worth investigating. Set up email alerts in Search Console for new coverage issues so you are notified of crawl problems before they affect rankings.
Crawl Errors vs Coverage Errors: The Difference
Crawl errors and coverage errors are related but distinct concepts in Google Search Console, and they appear in different reports.
Crawl errors are access-level failures — problems that prevent Googlebot from fetching a URL at all. These include 4xx not-found responses, 5xx server errors, DNS resolution failures, and connection timeouts. They are found in the Crawl Stats report under Settings.
Coverage errors are indexing-decision problems — cases where Googlebot successfully fetched the page but Google decided not to index it. Coverage errors appear in the Pages report under Indexing. Examples include pages with a noindex tag, pages where Google selected a different canonical, pages marked as "Crawled — currently not indexed" due to quality signals, and pages blocked by robots.txt (which shows as "Excluded by robots.txt").
The practical difference: crawl errors mean Google cannot read your page at all. Coverage errors mean Google read the page and decided not to include it in the index. Both need fixing, but the diagnostic process is entirely different. Start with crawl errors — if Google cannot reach a page, nothing else matters.
Crawl Error Impact on Rankings
Crawl errors have both direct and indirect effects on your search performance. The direct effect is straightforward: a page that cannot be crawled cannot be indexed, and a page that cannot be indexed cannot rank.
The indirect effects are less obvious but equally important:
- Wasted crawl budget — every 404 and 5xx response wastes a portion of the crawl budget Google allocates to your site. Fewer errors means more of that budget goes toward crawling your good pages.
- Server error escalation — when Googlebot repeatedly encounters 5xx errors, it interprets this as a sign that your server is overloaded. It responds by reducing the crawl rate for your entire domain, not just the affected URLs. This can cause a significant drop in how frequently your site is crawled.
- Sitemap 404s signal poor maintenance — URLs in your sitemap that return 404 are a strong signal that your sitemap is not being maintained. This can reduce Google's trust in your sitemap, causing it to rely more on link discovery instead.
- Internal link equity leakage — internal links pointing to pages that return errors pass no PageRank. This reduces the effectiveness of your site's internal link structure.
Crawl Error Fix Priority Order
Not all crawl errors are equally urgent. Fix them in this order:
- 1. Remove sitemap 404s — URLs in your sitemap that return 404 are the highest priority. Either restore the page, redirect the URL, or remove it from your sitemap. This is the most direct signal of a poorly maintained site structure.
- 2. Fix 5xx server errors — server errors affect Googlebot's crawl rate for your entire domain. Identify the root cause (database issues, memory limits, timeout configurations) and fix them immediately. Monitor server logs during high-traffic periods.
- 3. Fix DNS and connectivity issues — if Googlebot cannot resolve your domain or connect to your server, your entire site becomes inaccessible. These are rare but catastrophic when they occur.
- 4. Review 403 access blocks — check whether Googlebot is being blocked by your CDN, security plugin, or server configuration. Verify Googlebot IP ranges are whitelisted.
- 5. Update internal links pointing to redirects — links pointing to URLs that redirect (3xx) waste crawl budget and dilute PageRank. Update them to point directly to the final destination URLs.