Crawl budget is a favourite explanation when a site is underperforming, partly because it sounds technical and partly because it is hard to disprove. This article sets out what the documentation actually says about who needs to think about crawl budget, what shapes it, what wastes it and where to look for evidence.
Who the crawl budget guide is written for
Google’s crawl budget guide opens by saying that if a site does not have a large number of pages that change rapidly, or if its pages seem to be crawled the same day they are published, the guide is not needed. For Google Search, it says, keeping the sitemap up to date and checking the Page Indexing report regularly is adequate.
It describes itself as an advanced guide intended primarily for three kinds of site: large sites of 1 million or more unique pages with content that changes about once a week; medium or larger sites of 10,000 or more unique pages with very rapidly changing content, meaning daily; and sites with a large portion of their URLs classified in Search Console as Discovered – currently not indexed. Google adds that the numbers are a rough estimate to help classify a site and are not exact thresholds.
Our reading, not Google’s wording: this makes crawl budget a question to answer with evidence before it becomes a diagnosis. A site of a few thousand pages with a stable catalogue is, on Google’s own description, outside the target audience. If its pages are not being indexed, the cause is more likely to be found in indexing, duplication or quality than in how much Google crawls.
Capacity and demand: the two halves of a budget
Google’s documentation says that the web is a nearly infinite space and that there are limits to the time and resources it can devote to crawling any single site. The allocation is commonly called crawl budget. In this context a site is a unique hostname, so www.example.com and code.example.com have separate budgets. Two elements determine it.
The first is the crawl capacity limit, also known as hostload. It limits the total time your server spends holding connections open for Google. Every site starts with the same conservative default, and Google adjusts it over time if there is demand and the site stays healthy. If the site responds consistently and quickly, the limit goes up. If it slows down, returns 5xx server errors, or sends rate-limiting signals such as HTTP 429, the limit goes down and Google crawls less.
The second is crawl demand. For Googlebot it varies with a site’s size, update frequency, page quality and relevance compared with other sites. The guide names three factors: perceived inventory, popularity and staleness. Of these it calls perceived inventory the factor you can positively control the most. Without guidance, Google tries to crawl all or most of the URLs it knows about, so if many are duplicates or otherwise unwanted, that wastes a lot of crawling time.
Put together, crawl budget is the set of URLs Google can and wants to crawl. Even if the capacity limit is not reached, low demand means Google will crawl a site less. Not every crawled page is indexed either: after crawling, each page must be evaluated, consolidated and assessed for suitability for the index.
What wastes crawling on a large site
The guide’s best practices are mostly about inventory. Taken in turn:
- Duplicate content. Consolidate it, so that crawling focuses on unique content rather than unique URLs.
- Unimportant URLs. If duplicates cannot be consolidated, block them with robots.txt. The guide specifically says not to use noindex for this, because Google still requests the page and then drops it, wasting crawling time.
- Removed pages. Return 404 or 410 for permanently removed pages. A 404 is a strong signal not to crawl the URL again, while blocked URLs stay in the crawl queue for much longer.
- Soft 404s. These pages continue to be crawled and waste budget. The guide points to the Page Indexing report for finding them.
- Long redirect chains. The guide says to avoid them because they have a negative effect on crawling.
- Stale sitemaps. Google reads sitemaps regularly, so they should include everything you want crawled, and the guide recommends including lastmod for updated content.
One caution from the guide deserves emphasis. Do not use robots.txt to temporarily reallocate crawl budget to other pages. Google will not shift the freed budget unless it is already hitting your site’s crawl capacity limit.
Faceted navigation, the classic budget drain
Google has a separate page on managing crawling of faceted navigation URLs. It says the common parameter-based implementation can generate infinite URL spaces, which harms a site in two ways. Overcrawling happens because crawlers cannot tell whether faceted URLs are useful until they have fetched them. Slower discovery follows, because time spent on useless URLs is time not spent on new, useful ones.
Its recommendations split by intent. If the faceted URLs do not need to be indexed, prevent crawling, for example with robots.txt rules, or by applying filters through URL fragments, which Google generally does not support in crawling and indexing. It describes rel=canonical and rel=nofollow as generally less effective in the long term, and notes that nofollow only works if every anchor to a given URL carries it. If they do need to be indexed, the page recommends the standard & separator, a fixed order for filters in the path, and a real 404 status for filter combinations that return no results.
Reading the Crawl Stats report
The evidence for crawl questions is in the Search Console Crawl Stats report. Google describes it as showing statistics about its crawling history: how many requests were made and when, what the server responded, and any availability problems. It is aimed at advanced users, and Google says a site with fewer than a thousand pages should not need it.
Three details from the page change how it should be read. URLs are the actual URLs requested, not assigned to canonical URLs. Each request in a server-side redirect chain counts as a separate request, so a chain inflates the totals. And some requests might not be counted, so the figures can differ slightly from server logs. The report breaks crawling down by total requests, download size, average response time, host status, response codes, file type, crawl purpose and Googlebot type.
Our reading, not Google’s wording: response codes and host status are worth checking first, because they feed the capacity limit, then the example URLs, to see what Google is spending time on.
Where this fits at Cultured Digital: large sites and crawl questions
Technical SEO is our flagship service, and large-site diagnostics are within it. On the technical SEO page, the crawl step asks whether a page is allowed and affordable to crawl, covering robots.txt, status codes, crawl budget and logs. Large website analysis, faceted navigation and parameters belong to the group described in our architecture guide, and robots.txt, sitemaps and log-file analysis to the crawling and indexing guide.
Tools report symptoms; the work is finding which step is failing, and why. We do not promise rankings or traffic numbers. We set out the opportunity and the evidence, and measure against it.