Crawl Budget: What It Is and How to Optimize It
What is crawl budget?
Crawl budget is the set of URLs Google can and wants to crawl on your site. It combines crawl capacity (how much crawling your server can handle) and crawl demand (how much Google wants your pages).
- •Google's guide is for very large sites, fast-changing sites, and sites with many “Discovered” URLs.
- •Duplicates, soft 404s and long redirect chains waste it.
- •A noindex tag doesn't save it; robots.txt and real 404s do.
- •More budget comes from a healthier server and more valuable content.
Google can’t crawl every URL on the web as often as it would like, so it sets limits per site. Its crawl budget guide defines a site’s budget as the set of URLs Google can and wants to crawl. Both halves matter: a fast server doesn’t help if Google has little interest in your pages, and strong demand doesn’t help if your server can’t keep up.
Do You Need to Worry About It?
Most sites don’t. Google’s guide says it’s written mainly for three kinds of sites:
- Large sites, about 1 million or more unique pages, with content that changes around once a week.
- Medium or larger sites, 10,000 or more unique pages, with content that changes daily.
- Sites where a large share of URLs show as “Discovered - currently not indexed” in Search Console.
Google calls these numbers rough estimates, not exact thresholds. Search Console’s Crawl Stats help is blunter: if your site has fewer than a thousand pages, you shouldn’t need that report at all. For a small site that isn’t getting indexed, the cause is usually something else, such as content quality, duplicates or a technical block.
How Crawl Budget Works
Crawl capacity: how much your server can take
Google limits crawling so it doesn’t overload your server. Every site starts with the same conservative limit. If your server responds steadily and its response times hold or improve, the limit rises. If it slows down or returns 5xx errors or 429 “too many requests” responses, the limit drops. The limit is shared by all of Google’s crawlers, and each hostname counts as its own site: www.example.com and shop.example.com are budgeted separately.
Crawl demand: how much Google wants your pages
For Googlebot, demand depends on the site’s size, how often it updates, page quality and relevance compared with other sites. The three factors you can influence are your URL inventory (duplicates and unwanted URLs waste crawl time), popularity (URLs that are more popular on the web are crawled more often) and staleness (Google recrawls pages often enough to catch changes). Google says inventory is the factor you can improve the most.
Everything Googlebot fetches counts, not just pages. Google has said alternate URLs (such as AMP and hreflang versions) and embedded resources like CSS, JavaScript and API (XHR) calls can consume crawl budget too.
How to Check Your Crawl Budget
There’s no single “budget” number to look up, but Search Console shows how Google actually crawls you. Open Settings > Crawl stats (it’s available only for root-level properties, such as a Domain property or https://example.com). The report shows:
- Total crawl requests and download size over time. A sudden drop is worth investigating.
- Average response time. Rising times shrink your crawl capacity.
- Host status: problems with robots.txt fetching, DNS or server connectivity.
- Crawl responses: how many requests got 200, 301, 404, 5xx and so on. A high share of redirects and errors means crawls spent on URLs you don’t want.
- Crawl purpose: “Discovery” is a URL Google never crawled before; “Refresh” is a recrawl. If changing pages aren’t refreshed often enough, the help suggests making sure they’re in your sitemap.
Two more checks: in URL Inspection, a “Hostload exceeded” message means your server’s capacity is the limit. And in the Page indexing report, a large and growing “Discovered - currently not indexed” count is the classic budget symptom. For URL-level detail, your server logs show every Googlebot request; see log file analysis.
How to Optimize Crawl Budget
These are the practices in Google’s guide, in roughly the order they tend to pay off:
1. Consolidate duplicate content
Several URLs with the same content split crawling across copies. Merge them into one page and redirect the rest, so Google spends its visits on unique content rather than unique URLs.
2. Block unimportant URL spaces with robots.txt
Some pages are fine for users but not worth crawling: sorted versions of the same list, endless filter combinations, infinite-scroll duplicates, internal search results. If you can’t consolidate them, block them. See faceted navigation for filter-heavy stores.
User-agent: * Disallow: /search Disallow: /*?sort= Disallow: /*&sort=
Test patterns against real URLs before you ship them. And two things Google says not to do: don’t use noindex to save budget (Google still requests the page, then drops it), and don’t block URLs temporarily hoping Google will spend the budget elsewhere. It won’t, unless your site is already at its capacity limit.
3. Return 404 or 410 for pages that are gone
Google never fully forgets a URL, but a 404 is a strong signal not to crawl it again. Blocked URLs, by contrast, stay in the crawl queue much longer. Also fix soft 404s: pages that say “not found” with a 200 keep getting crawled.
4. Keep sitemaps current
Google reads your sitemap regularly. List all the content you want crawled, only in its final URL, and use <lastmod> for pages that change. Take out redirects, 404s and noindexed pages.
5. Avoid long redirect chains
Every hop is another request. Point links and redirects straight at the final URL. See redirect chains.
6. Make pages cheap to load
If Google can load and render pages faster, it may read more of your site. Improve server response times, trim heavy resources, and support HTTP caching: answering a conditional request with 304 Not Modified tells Google to reuse its cached copy. You can check it:
# 1. Read the page's Last-Modified value curl -sI https://www.example.com/page | grep -i last-modified # 2. Send it back. A 304 means conditional requests work. curl -sI https://www.example.com/page -H 'If-Modified-Since: <value from step 1>' | head -1
How to Get More Crawl Budget
Google’s guide names two ways. Add server resources if capacity is the limit, for example when URL Inspection reports “Hostload exceeded.” And improve content quality for the product you care about: for Google Search, Google lists popularity, overall user value, content uniqueness and serving capacity. In other words, the lasting way to be crawled more is to be worth crawling more.
Outdated Advice to Ignore
- “Set your crawl rate in Search Console.” Google removed the crawl rate limiter tool on January 8, 2024. Googlebot now adjusts on its own based on how your server responds.
- “Use the URL Parameters tool.” Google retired it in 2022. Use robots.txt, consistent internal links and canonical URLs instead.
- “Add crawl-delay to robots.txt.” Google has said Googlebot doesn’t process the non-standard crawl-delay rule.
- “Canonical tags save crawl budget.” A canonical helps Google choose which URL to index, but Google has to crawl a page to read its canonical tag. It doesn’t stop crawling by itself.
FAQ
Does crawl budget affect rankings?
Not directly. Google has said that crawling is necessary to appear in results but is not a ranking signal, and that a higher crawl rate won’t necessarily improve positions. Budget matters because pages Google doesn’t crawl can’t be indexed or updated.
How do I check how many pages Google crawls per day?
Search Console’s Crawl stats report shows total crawl requests by day. Server logs filtered to Googlebot give the exact URLs.
Does nofollow save crawl budget?
Not reliably. Google has said any URL it crawls counts, so a nofollowed link can still be crawled if Google finds the URL elsewhere.
Do small sites need to optimize crawl budget?
Rarely. Google’s guide targets very large or fast-changing sites. On a small site, spend the effort on content and internal links instead.