# How do you identify URLs that unnecessarily consume crawler resources?

> Classify URLs by type and compare crawl activity against SEO value.

Look for:

- parameter URLs;

- duplicate URLs;

- unnecessary sorting/filtering combinations;

- internal search URLs;

- session URLs;

- redirect chains;

- soft 404s;

- infinite URL spaces;

- obsolete URLs;

- dynamically generated low-value URLs.

Use server logs to identify how often Googlebot requests each pattern.

A useful analysis is:

**crawl requests by URL type ÷ total Googlebot requests**

Then ask whether the percentage spent on each type is justified.

Google recommends managing URL inventory, consolidating duplicate content, and blocking unnecessary URLs where appropriate.

## Sources

- [Google Search Central — Crawl Budget](https://developers.google.com/crawling/docs/crawl-budget)

## Related questions

- [Can blocking a URL in robots.txt solve an indexation problem?](https://www.theseocentral.com/technical-seo-faq/can-blocking-a-url-in-robots-txt-solve-an-indexation-problem.md)
- [How do you analyze Googlebot crawl behavior beyond Search Console?](https://www.theseocentral.com/technical-seo-faq/how-do-you-analyze-googlebot-crawl-behavior-beyond-search-console.md)
- [How do you decide which URL patterns should be crawlable?](https://www.theseocentral.com/technical-seo-faq/how-do-you-decide-which-url-patterns-should-be-crawlable.md)
- [How do you determine whether a website actually has a crawl-budget problem?](https://www.theseocentral.com/technical-seo-faq/how-do-you-determine-whether-a-website-actually-has-a-crawl-budget-problem.md)
- [How do you differentiate crawl demand from crawl capacity?](https://www.theseocentral.com/technical-seo-faq/how-do-you-differentiate-crawl-demand-from-crawl-capacity.md)
- [How would you handle millions of faceted-navigation URLs?](https://www.theseocentral.com/technical-seo-faq/how-would-you-handle-millions-of-faceted-navigation-urls.md)
- [How would you optimize crawling on a website with millions of URLs?](https://www.theseocentral.com/technical-seo-faq/how-would-you-optimize-crawling-on-a-website-with-millions-of-urls.md)
- [What patterns in server logs indicate inefficient crawling?](https://www.theseocentral.com/technical-seo-faq/what-patterns-in-server-logs-indicate-inefficient-crawling.md)
- [When would you use robots.txt versus noindex?](https://www.theseocentral.com/technical-seo-faq/when-would-you-use-robots-txt-versus-noindex.md)

---

Answered by Abhinav Krishna (https://www.theseocentral.com/author/abhinav-krishna)
Source: https://www.theseocentral.com/technical-seo-faq/how-do-you-identify-urls-that-unnecessarily-consume-crawler-resources