Crawl Budget

Search engines send automated bots (crawlers) to fetch your pages before they can rank them. Crawl budget is simply the practical ceiling on how many of your URLs Googlebot will fetch in a given period. It is governed by two things: how hard Google can hit your server without slowing it down (crawl rate limit), and how much Google actually wants your content (crawl demand, driven by popularity and freshness).

For most sites under a few thousand URLs, crawl budget is a non-issue — do not waste time on it. It becomes real on large sites where thin, duplicate, or parameter URLs soak up fetches that should go to money pages. Diagnose it in Search Console (Crawl Stats and the Pages report) and your server logs: look for Googlebot spending its time on faceted-navigation URLs, session IDs, or infinite calendars. The fix is almost always subtraction — noindex or block the junk, consolidate duplicates, flatten deep click paths, and keep your sitemap clean.

At enterprise scale crawl budget is really a site-architecture and information-quality problem wearing a technical mask. If Google is not crawling your important pages fast enough, the answer is rarely "ask for more crawl" — it is to stop diluting demand across low-value URLs and to raise the average quality signal of what you expose. Watch crawl efficiency (share of crawls hitting indexable, canonical, 200 URLs) as your KPI, and treat log-file analysis as a first-class discipline, not an afterthought.