Crawl budget: definition

SEO

By Alvyn Detavernier, founder · published October 28, 2026

Definition

Crawl budget is the set of URLs a search engine can and wants to crawl on a website within a given period. It depends on how much load the server can handle without slowing down and on how much Google wants to revisit the pages, based on their popularity and how stale its copies are.

Also known as: crawl capacity and demand, Googlebot crawl budget

Crawl capacity and crawl demand

Google combines two ideas. The crawl capacity limit is how many parallel connections and how much server time Googlebot can use without overloading your site: it rises when the server responds quickly and drops after 5xx errors or 429 rate-limit responses. Crawl demand is how much Google wants to visit your pages: it depends on how many URLs it knows, how popular they are and how long since its last visit. A site migration raises demand temporarily.

Who should actually care

Google points to three profiles: large sites with around a million unique pages that change weekly, sites with around 10,000 pages that change daily, and sites with many URLs stuck as "Discovered, currently not indexed" in Search Console. A 40-page business website whose new pages get crawled the same day has no crawl budget problem.

The most common crawl waste

Faceted navigation creating thousands of filter URLs: block those parameters in robots.txt.
Duplicate content that was never consolidated.
Deleted pages returning 200 (soft 404s) instead of 404 or 410.
Long redirect chains.
Noindexed pages: Google still has to fetch them, so noindex does not save budget.
Sitemaps listing stale URLs or without a trustworthy lastmod.

How to measure crawling

The Crawl stats report, under Settings in Search Console, shows requests per day, average response time and breakdowns by status code and file type. For deeper analysis, server logs reveal which sections Googlebot actually visits: it is the only way to know whether your product pages get crawled or the bot burns its visits on sort orders. A fast server that supports 304 Not Modified responses also lets Google reuse its cached copies.

Example

Example: a store with 3,000 products generates 250,000 filter URLs for color, size and price. Log analysis shows 70% of Googlebot hits land on those filters. After blocking the parameters in robots.txt, new product pages get crawled within two days instead of three weeks.

Sources

Crawl Budget Management For Large Sites, Google Search Central

Frequently asked questions.

Does my small website have a crawl budget problem?

Almost certainly not. Google reserves the topic for sites with tens of thousands of frequently changing pages. When a page on a small site is not indexed, the cause is usually quality, duplication, a stray noindex tag or missing internal links, not a lack of crawl budget.

Does noindex save crawl budget?

No. To read a noindex tag, Googlebot has to download the page, which uses budget on every visit. To cut crawling of useless pages, block their paths in robots.txt, remove them with a 404 or 410 status, or consolidate them with a canonical tag or a redirect.

Related terms.

The concepts that go with it, to round out the definition.

SEO

Robots.txt

Robots.txt is a plain text file at the root of a website that tells crawlers which areas they may or may not visit. It manages crawler traffic rather...

SEO

XML sitemap

An XML sitemap is a file that lists the URLs on a website you want search engines to crawl and index, optionally with the date each page was last...

SEO

Indexing

Indexing is the stage where a search engine analyzes a page it has crawled and stores it in its index, the database it draws search results from. A...

SEO

301 redirect

A 301 redirect is a server response stating that a page has permanently moved to a new address. Browsers and crawlers are sent to the new URL...

SEO

404 error

A 404 error (Not Found) is the HTTP status code a server returns when the requested page cannot be found, for example after it was deleted or the...

Put it into practice.

How we can help

Web design

Glossary

Browse the full glossary

Digital marketing terms, from A to Z.

A second pair of eyes

Does this apply to your business? Let's look at your numbers.

In 30 minutes, we look at how this applies to your business, with your real data. Free, no commitment: you leave with a plan, even if we never work together.

Book a free call