Crawl budget — the number of pages a search engine is willing to crawl on a given site in a given period — mostly matters for larger sites, typically several thousand pages or more. On a small site, crawl budget is rarely the actual bottleneck; this is worth ruling out first before spending effort here.
How to tell if crawl budget is actually a problem for you
- Google Search Console's Crawl Stats report shows a large share of requests going to low-value URLs (filtered/sorted listing variants, tag pages, session parameters).
- Important pages sit in "Discovered, not indexed" for weeks despite being linked internally and included in the sitemap.
- The site has grown significantly in page count faster than its internal linking and authority have grown to support it.
Fix 1 — Eliminate crawlable, low-value URL variants
Faceted navigation, sort/filter parameters, and session IDs can generate enormous numbers of near-duplicate, crawlable URLs. Block the low-value parameter patterns in robots.txt or consolidate them with canonical tags so crawlers stop repeatedly requesting pages that add no unique value.
Fix 2 — Fix redirect chains
Every redirect hop consumes crawl budget and delays a crawler reaching the actual destination page. This is one of the most common crawl-budget drains after a site migration that wasn't fully cleaned up — flatten any A→B→C chain down to a single direct redirect.
Fix 3 — Reduce soft 404s and broken internal links
Crawlers keep re-requesting URLs that return a "page not found" message with a 200 status code (a soft 404), and broken internal links waste crawl requests on dead ends. Both should be found in a crawl report and either fixed or given a proper 404/410 status.
Fix 4 — Consolidate near-duplicate or thin pages
Multiple thin pages targeting near-identical topics split both crawl budget and ranking signals. Consolidating them into one stronger page (with a redirect from the removed URLs) usually improves both crawl efficiency and rankings simultaneously.
Fix 5 — Serve a lean, accurate XML sitemap
The sitemap should list only real, canonical, indexable URLs — not redirected, noindexed, or parameter-variant pages. On very large sites, splitting into multiple sitemaps by section keeps each one manageable and easier to monitor in Search Console.
Fix 6 — Server response time
A slow server response reduces how many pages a crawler will fetch in a given crawl session — crawlers throttle back when a site responds slowly, which is as much an IT infrastructure issue as an SEO one.
| Crawl Budget Drain | Fix |
|---|---|
| Faceted navigation / parameter URLs | Robots.txt disallow or canonical consolidation |
| Redirect chains | Flatten to a single direct redirect |
| Soft 404s / broken internal links | Fix or return proper 404/410 status |
| Thin, near-duplicate pages | Consolidate into one stronger page with a redirect |
| Slow server response time | Improve hosting/server performance |
Crawl budget fixes rarely move rankings directly — what they do is make sure the pages you actually want indexed and updated are the ones getting a search engine's attention, which is foundational to any serious SEO & Search Growth program on a larger site.
FAQ
What is crawl budget in SEO?
Crawl budget is the number of pages a search engine is willing and able to crawl on a given site within a given period, and it mainly becomes a limiting factor on larger sites (typically several thousand pages or more) where wasted crawler requests on low-value URLs can delay important pages from being crawled and re-indexed.
- Small sites rarely have a genuine crawl budget problem — it's worth confirming the issue exists before investing effort here.
- Signs of a real problem include Crawl Stats showing heavy requests to low-value URLs and important pages stuck in "Discovered, not indexed."
How do you fix crawl budget issues on a large website?
Fix crawl budget issues by blocking or consolidating low-value crawlable URL variants (faceted navigation, parameters), flattening redirect chains to a single hop, fixing soft 404s and broken internal links, consolidating thin near-duplicate pages, keeping the XML sitemap limited to real canonical URLs, and improving server response time.
- Redirect chains left over from a past site migration are one of the most common and fixable crawl budget drains.
- Server response time affects crawl budget directly — slower servers get crawled less per session.