Technical SEO FAQ · Crawling, Crawl Budget & URL Management
How do you differentiate crawl demand from crawl capacity?
Google describes crawl budget using two major concepts:
Crawl capacity limit
How much Google can crawl without overwhelming the site.
Crawl demand
How much Google wants to crawl.
A site can have high theoretical capacity but low crawl demand. Increasing server capacity would not necessarily cause Google to crawl more.
Conversely, if server performance deteriorates, Google can reduce crawling to protect the site.
Site changes such as migrations can increase crawl demand because Google needs to process new URLs and changes.
Related questions
- Can blocking a URL in robots.txt solve an indexation problem?
- How do you analyze Googlebot crawl behavior beyond Search Console?
- How do you decide which URL patterns should be crawlable?
- How do you determine whether a website actually has a crawl-budget problem?
- How do you identify URLs that unnecessarily consume crawler resources?
- How would you handle millions of faceted-navigation URLs?
- How would you optimize crawling on a website with millions of URLs?
- What patterns in server logs indicate inefficient crawling?
- When would you use robots.txt versus noindex?