# How do you determine whether a website actually has a crawl-budget problem?

> Do not infer a crawl-budget problem from URL count alone.

Google's current crawl-budget guidance is primarily aimed at very large or rapidly changing sites, including roughly 1 million+ unique pages with moderate change, 10,000+ pages changing very rapidly, or sites with a large share of URLs reported as "Discovered - currently not indexed." These are rough guidance thresholds, not hard limits.

Investigate:

- Crawl Stats in Search Console.

- Server logs.

- URL inventory.

- Crawl distribution by URL type.

- Important URLs that remain undiscovered or stale.

- Server response capacity.

- Duplicate and parameter URL volume.

The strongest evidence is not "Googlebot crawled 500,000 URLs." It is evidence that crawling is inefficient or constrained in a way that affects important URLs.

## Sources

- [Google Search Central — Crawl Budget](https://developers.google.com/crawling/docs/crawl-budget)

## Related questions

- [Can blocking a URL in robots.txt solve an indexation problem?](https://www.theseocentral.com/technical-seo-faq/can-blocking-a-url-in-robots-txt-solve-an-indexation-problem.md)
- [How do you analyze Googlebot crawl behavior beyond Search Console?](https://www.theseocentral.com/technical-seo-faq/how-do-you-analyze-googlebot-crawl-behavior-beyond-search-console.md)
- [How do you decide which URL patterns should be crawlable?](https://www.theseocentral.com/technical-seo-faq/how-do-you-decide-which-url-patterns-should-be-crawlable.md)
- [How do you differentiate crawl demand from crawl capacity?](https://www.theseocentral.com/technical-seo-faq/how-do-you-differentiate-crawl-demand-from-crawl-capacity.md)
- [How do you identify URLs that unnecessarily consume crawler resources?](https://www.theseocentral.com/technical-seo-faq/how-do-you-identify-urls-that-unnecessarily-consume-crawler-resources.md)
- [How would you handle millions of faceted-navigation URLs?](https://www.theseocentral.com/technical-seo-faq/how-would-you-handle-millions-of-faceted-navigation-urls.md)
- [How would you optimize crawling on a website with millions of URLs?](https://www.theseocentral.com/technical-seo-faq/how-would-you-optimize-crawling-on-a-website-with-millions-of-urls.md)
- [What patterns in server logs indicate inefficient crawling?](https://www.theseocentral.com/technical-seo-faq/what-patterns-in-server-logs-indicate-inefficient-crawling.md)
- [When would you use robots.txt versus noindex?](https://www.theseocentral.com/technical-seo-faq/when-would-you-use-robots-txt-versus-noindex.md)

---

Answered by Abhinav Krishna (https://www.theseocentral.com/author/abhinav-krishna)
Source: https://www.theseocentral.com/technical-seo-faq/how-do-you-determine-whether-a-website-actually-has-a-crawl-budget-problem