What Is a Crawl Budget, in Google's Words?
The term is Google's, and Google has defined it twice. In 2017, Gary Illyes wrote on the Google Search Central blog that “we don't have a single term that would describe everything that ‘crawl budget’ stands for externally”, then gave one: “Taking crawl rate and crawl demand together we define crawl budget as the number of URLs Googlebot can and wants to crawl” (Google, 2017).
The current guide, now hosted in Google's crawling infrastructure documentation and last updated July 22, 2026, says: “there are limits to how much time and resources Google can devote to crawling any single site. The allocation of these resources is commonly called a site's crawl budget.” It concludes: “Taking crawl capacity and crawl demand together, Google defines a site's crawl budget as the set of URLs that Google can and wants to crawl” (Google).
Crawl Budget at a Glance
| Question | Answer | Source |
|---|---|---|
| Who defines it | Google; the term is used informally for other crawlers too | |
| Two parts | Crawl capacity limit (how much your server can take) and crawl demand (how much Google wants) | |
| Unit of a “site” | A unique hostname; www and a subdomain have separate budgets | |
| Who needs to care | Roughly 1 million+ pages changing weekly, or 10,000+ changing daily | |
| Where to see it | Search Console Crawl Stats report | Search Console Help |
| Ranking factor? | No: Google says “it's not a ranking signal” | Google, 2017 |
| Related terms | Crawl rate, hostload, indexing, Discovered – currently not indexed | See table below |
The Two Parts: Capacity and Demand
Crawl capacity limit. Google says its crawlers “calculate a crawl capacity limit (also known as hostload)” that “limits the total amount of time your server spends holding connections open for Google, factoring in both the number of parallel connections and their duration.” Every site “starts with the same default, conservative crawl capacity limit”, which rises if the site responds consistently and falls if it slows down or returns server errors (5xx) or rate-limiting signals such as HTTP 429.
Crawl demand. For Googlebot, Google says demand “varies based on a site's size, update frequency, page quality, and relevance, compared to other sites.” The factors it says you can influence are perceived inventory (“the factor that you can positively control the most”), popularity and staleness. Site moves can raise demand temporarily.
The current guide adds something the 2017 post did not stress: “While each crawler has a different crawl demand, the crawl capacity limit is shared across all crawlers. This means that high demand from one crawler can reduce the capacity available for others.” AdsBot and Google Shopping crawlers draw on the same capacity as Googlebot. It also notes that crawling does not guarantee indexing: “not every page that is crawled will necessarily be indexed.”
Who Needs to Worry About Crawl Budget
Most sites do not. The guide opens: “If your site doesn't have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don't need to read this guide.” For those sites it says keeping the sitemap up to date and checking the Page Indexing report is adequate. The 2017 post put it more bluntly: “if a site has fewer than a few thousand URLs, most of the time it will be crawled efficiently.”
Unique pages; Google calls these “a rough estimate”, not exact thresholds
Source: Google crawling documentation, Optimize your crawl budget, read October 10, 2026.
A third group in the guide has no size figure: sites with a large share of URLs reported as Discovered – currently not indexed. The 10,000 bar is 1% of the 1,000,000 bar, drawn to scale.
The third group, sites where many URLs sit in “Discovered – currently not indexed”, is the one smaller sites sometimes fall into. Our step-by-step guide to fixing Discovered – currently not indexed covers that status, and how long Google takes to index a new page sets out what Google says about timing.
What Wastes Crawl Budget
The 2017 post listed low-value URL types that, in Google's analysis, “can negatively affect a site's crawling and indexing”, “in order of significance”:
- Faceted navigation and session identifiers
- On-site duplicate content
- Soft error pages
- Hacked pages
- Infinite spaces and proxies
- Low quality and spam content
The current guide turns this into practice. It recommends consolidating duplicates, blocking unimportant URLs such as sorted or infinitely scrolling variants with robots.txt, returning 404 or 410 for removed pages, eliminating soft 404s (“soft 404 pages will continue to be crawled, and waste your budget”), keeping sitemaps current with <lastmod>, avoiding long redirect chains and supporting 304 (Not Modified) responses. On duplicates, see canonical tag vs noindex; on chains, 301 vs 302 redirects.
Two of its warnings run against common advice. On noindex: “Don't use noindex, as Google will still request, but then drop the page.” On robots.txt as a dial: “Google won't shift this newly available crawl budget to other pages unless Google is already hitting your site's crawl capacity limit.”
Limits Per Fetch
Crawl budget is about how many URLs Google fetches; a separate limit governs how much of each file it reads. Google's Googlebot documentation says that for Google Search, “Googlebot crawls the first 2MB of a supported file type, and the first 64MB of a PDF file.” It also says that “For most sites, Googlebot shouldn't access your site more than once every few seconds on average.”
Megabytes, uncompressed
Source: Google Search Central, Googlebot, last updated February 3, 2026, read October 10, 2026.
Once the cutoff is reached, Googlebot stops the fetch and sends only the part already downloaded for indexing. Each CSS and JavaScript resource is fetched separately under the same limit. Other Google crawlers may have different limits.
Redirects add requests as well. Google's status code reference says its crawlers follow up to 10 redirect hops, and the Crawl Stats help says “each request in the redirect chain is counted as a separate request.”
How to See Your Crawl Budget
Google does not publish a number for any site. The closest view is the Search Console Crawl Stats report, which the help page says shows total crawl requests, total download size, average response time, host status, and breakdowns by response, file type, crawl purpose and Googlebot type. The same page says “All URLs shown and counted are the actual URLs requested by Google”, that host status covers the past 90 days, and that a known issue means “some requests might not be counted.” If you have not set up Search Console yet, start with verifying your site in Google Search Console.
What to look for, according to that page: “Most responses should be 200 or other ‘Good’-type responses, unless you are doing a site reorganization or site move.” Rising response times, server errors or “robots.txt not available” are the signs that capacity is the constraint.
How to Get More Crawl Budget
Google names two ways. First, “Add more server resources” if your server is the limit, for example when the URL Inspection tool reports “Hostload exceeded”. Second, “Optimize your content's quality for the Google product you're targeting”, which for Search includes “popularity, overall user value, content uniqueness, and serving capacity” (Google). The 2017 post added that faster sites get more crawling: “a speedy site is a sign of healthy servers, so it can get more content over the same number of connections.” Links from other sites are part of popularity; see what a backlink is.
Crawl Budget vs Related Terms
| Term | What it is | Relationship to crawl budget |
|---|---|---|
| Crawl capacity limit (hostload) | The ceiling Google sets so it does not overload your server | One of the two parts of crawl budget |
| Crawl demand | How much Google wants to crawl, from inventory, popularity and staleness | The other part |
| Crawl rate | Google's 2017 term for the fetching limit; also used loosely for requests per second | Older name for capacity |
| Indexing | Storing a page so it can appear in results | Separate step; crawled pages are not always indexed |
| robots.txt | File that tells crawlers which URLs not to fetch | A tool for managing inventory; see noindex vs robots.txt |
| Robots meta tag | Page-level indexing and snippet rules | Read only after crawling; see what a robots meta tag is |
Common Misconceptions About Crawl Budget
| Myth | What the source actually says | Source |
|---|---|---|
| Every site should optimize crawl budget | Google says sites whose pages are crawled the same day they are published don't need to read its guide | |
| More crawling improves rankings | “An increased crawl rate will not necessarily lead to better positions”; Google says “it's not a ranking signal” | Google, 2017 |
| Noindex saves crawl budget | Google still requests the page, then drops it, “wasting crawling time” | |
| crawl-delay in robots.txt slows Googlebot | “The non-standard crawl-delay robots.txt rule is not processed by Googlebot” | Google, 2017 |
| Blocking pages frees budget for others | Only if Google is already hitting your capacity limit | |
| A 404 makes Google forget a URL | “Google won't forget a URL that it knows about”, but a 404 is a strong signal not to crawl it again |
Frequently Asked Questions
What is crawl budget in simple terms?
It is the set of URLs on your site that Google can and wants to crawl in a given period. It depends on how much load your server can take and how much Google wants your content.
Does crawl budget matter for small sites?
Usually not. Google says its guide is for very large or fast-changing sites, roughly 1 million or more pages changing weekly or 10,000 or more changing daily, and that sites crawled on the day they publish need not worry.
Is crawl budget a ranking factor?
No. Google's 2017 post says crawling is necessary for being in results but is not a ranking signal, and more crawling does not necessarily lead to better positions.
How do I check my crawl budget?
Google publishes no figure. The Search Console Crawl Stats report shows total crawl requests, download size, average response time, host status and responses by type, which together show how Google is crawling your site.
Does noindex save crawl budget?
No. Google says it still requests noindexed pages and then drops them, wasting crawling time. It recommends robots.txt for URLs you do not want crawled, and 404 or 410 for removed pages.
How do I increase my crawl budget?
Google names two routes: add server resources if your server is the bottleneck, and improve content quality, which for Search includes popularity, user value, uniqueness and serving capacity. Faster responses also help.
Do subdomains share a crawl budget?
No. Google defines a site as a unique hostname for crawling, so www.example.com and code.example.com have separate crawl budgets. The crawl capacity limit is shared across Google's crawlers for a given host.
Sources and Methodology
This explainer is built from Google's current crawl budget guide, its 2017 Search Central blog post, the Googlebot and HTTP status code documentation, and the Search Console Crawl Stats and Page indexing help pages, read on October 10, 2026. Definitions are quoted verbatim. Google publishes no crawl budget figure for any site, so we report none, and the site-size figures are Google's own rough estimates. We did not analyze server logs or test how crawling changes after any of the fixes described. Bing and other search engines run their own crawlers and are not covered here.
- official Optimize your crawl budget, Google crawling infrastructure documentation (last updated July 22, 2026). Read October 10, 2026.
- official What Crawl Budget Means for Googlebot, Google Search Central Blog, Gary Illyes, January 16, 2017. Read October 10, 2026.
- official Crawl Stats report, Search Console Help. Read October 10, 2026.
- official Googlebot, Google Search Central (last updated February 3, 2026). File size limits and access rate. Read October 10, 2026.
- official HTTP status codes, Google crawling documentation (last updated February 4, 2026). Redirect hops; 4xx and 5xx handling. Read October 10, 2026.
- official Page indexing report, Search Console Help. Read October 10, 2026.