Google Not Crawling My Site: How to Fix It

Confirm Google Is Not Crawling You

Before troubleshooting, confirm the problem is real. "Not crawling" can mean different things: Google has never crawled your site, Google stopped crawling, or Google is crawling but not indexing. Each has a different fix.

Three ways to verify crawl activity:

• GSC Crawl Stats — go to Settings → Crawl Stats in Google Search Console. This shows Googlebot crawl activity over the past 90 days. A flat line with zero requests means Googlebot has not been visiting your site.
• Server logs — filter your server access logs for the Googlebot user agent string: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html). No entries means no crawl. This is the most authoritative source because it cannot be spoofed by log filtering in third-party tools.
• URL Inspection — inspect a specific URL in Google Search Console. The "Coverage" section shows when Googlebot last crawled the URL and what it found.

Check Your robots.txt First

The most common cause of Googlebot not crawling a site is an overly restrictive robots.txt file. Navigate to https://yourdomain.com/robots.txt and look for Disallow: / — this single line blocks all crawlers from accessing any page on your site.

Platform-specific issues to watch for:

• WordPress maintenance mode — some maintenance plugins set a global Disallow in robots.txt while the site is in maintenance mode, then fail to remove it after the mode is deactivated.
• Squarespace and Wix password protection — enabling password protection on these platforms typically adds a Disallow: / to robots.txt automatically. Disabling protection removes it, but verify robots.txt manually.
• Staging sites that went live — staging environments often have robots.txt set to Disallow: / to prevent indexing. If a staging site was copied to production, this setting comes with it.

Use the robots.txt tester in Google Search Console (under Settings) to test specific URLs against your current robots.txt rules and confirm whether Googlebot is blocked.

Verify Googlebot Is Not Being Blocked by Hosting or CDN

Even when robots.txt is clean, Googlebot can be blocked at the server or CDN level. Common sources of this problem:

• Cloudflare Bot Fight Mode or Super Bot Fight Mode — these settings can block or challenge bots that Cloudflare identifies as automated traffic. Verified bots like Googlebot are typically exempt, but misconfigurations occur. Check Cloudflare's Security → Bots settings.
• WordPress security plugins — plugins like Wordfence, iThemes Security, or similar can block IP ranges they flag as suspicious. Googlebot's IP ranges sometimes trigger these filters.
• .htaccess IP restrictions — a deny from all or similar rule in .htaccess will block all traffic including Googlebot.

To confirm real Googlebot is being blocked: perform a reverse DNS verification. Real Googlebot IPs resolve to domains ending in googlebot.com or google.com. If your security tool is blocking the right IP ranges, ensure Googlebot's published IP ranges are whitelisted. Google publishes its crawler IP ranges at developers.google.com/search/apis/ipranges/googlebot.json.

Check if Your Site Returns Server Errors to Bots

Some server configurations or security plugins return 403 or 503 responses specifically for automated user agents, while serving normal 200 responses to human visitors. This makes the problem invisible to casual testing — you load the page fine in a browser, but Googlebot gets blocked.

Test this from the command line using curl with the Googlebot user agent:

curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -I https://yoursite.com/

Check the HTTP status code in the response. A 200 means the page is accessible to Googlebot. A 403 or 503 means your server is treating the Googlebot user agent as blocked traffic. If you see this, investigate your server-side bot filtering rules and check your server error logs for Googlebot-specific response patterns.

Submit or Resubmit Your XML Sitemap

Without a sitemap, Googlebot discovers new pages solely through link following. For sites with many pages or regularly added content, link discovery alone is slow and incomplete. A submitted sitemap gives Google a direct, explicit list of every URL you want crawled.

To submit your sitemap: open Google Search Console, go to Sitemaps under Indexing, and enter your sitemap URL (typically /sitemap.xml). After submission, GSC shows the fetch status — confirm it shows "Success" and not an error.

Common sitemap errors that prevent processing:

• Sitemap returns a non-200 HTTP status (404, 500)
• Sitemap is blocked by robots.txt (a Disallow: /sitemap.xml or similar rule)
• Sitemap contains malformed XML (missing closing tags, encoding errors)
• Sitemap URLs use a different domain than the Search Console property (domain mismatch)

Fix Slow Server Response Time

Googlebot adapts its crawl rate based on your server's response speed. When your server is slow, Googlebot reduces how aggressively it crawls to avoid overloading you. This is called crawl rate limiting. A Time to First Byte (TTFB) consistently over 500ms is enough to cause Googlebot to back off significantly.

To check server response time: use the Core Web Vitals report in GSC (which includes TTFB data), or test with PageSpeed Insights. Improvements that most reliably reduce TTFB:

• Enable server-side caching (Redis, Memcached, or full-page caching for WordPress)
• Use a CDN to serve static assets and cached pages from edge locations
• Optimize slow database queries — in WordPress, slow SQL queries from unoptimized plugins are a common TTFB killer
• Upgrade your hosting tier if the server is consistently at CPU or memory capacity

Build External Links to Trigger Crawling

Google allocates crawl budget based on a site's perceived importance, and PageRank from external links is a primary signal of importance. A brand-new site with zero external links gets minimal crawl budget — Googlebot may visit the homepage occasionally but will not crawl deep into the site.

The fastest shortcut to increased crawl frequency is getting a single link from an already-indexed, high-authority page. When Googlebot crawls that page and finds a link to your site, it follows it and allocates crawl budget accordingly. Tactics that work:

• Submit your site to relevant industry directories that are already indexed
• Publish a guest post on an established site in your niche with a link back
• Get listed in resource pages or tool roundups
• Share content on social platforms — while social links are typically nofollow, social traffic can lead to organic links from people who discover your content

Use the URL Inspection Request Indexing Feature

For specific high-priority pages, use URL Inspection in Google Search Console and click Request Indexing. This adds the URL to a priority crawl queue. It does not guarantee indexing, but it typically triggers a Googlebot visit within 24-48 hours.

Use this feature strategically — it is designed for individual high-priority URLs, not for re-indexing an entire site. If your crawl problem is sitewide, fixing the root cause (robots.txt, server blocking, sitemap errors) and resubmitting your sitemap is the correct approach. Requesting indexing for every URL manually will not solve a structural crawl block and will not scale.

After requesting indexing, check the URL status again in 24-72 hours. If the status remains "URL is not on Google" with the same blocking reason, revisit the diagnostic steps above — the root cause is still present.

• Sitemap Not Working
• Pages Not Indexed Google
• Sitemap Errors Search Console