# What patterns in server logs indicate inefficient crawling?

> Common patterns include:

- high request volume to parameter URLs;

- excessive redirect chains;

- repeated requests to obsolete URLs;

- large numbers of soft-404-like responses;

- repeated server errors;

- unnecessary URL variants;

- excessive crawling of low-value filters;

- inefficient cache behavior.

A 404 response itself is not automatically evidence of wasted crawl budget. Google's documentation notes that normal 4xx responses generally don't consume crawl budget in the same way as successful responses; 429 is an important exception.

Focus on patterns and consequences rather than treating every non-200 response as a problem.

## Sources

- [Google Search Central — Crawl Budget](https://developers.google.com/crawling/docs/crawl-budget)
- [Google Search Central — Myths About Crawling](https://developers.google.com/crawling/docs/myths-about-crawling)

## Related questions

- [Can blocking a URL in robots.txt solve an indexation problem?](https://www.theseocentral.com/technical-seo-faq/can-blocking-a-url-in-robots-txt-solve-an-indexation-problem.md)
- [How do you analyze Googlebot crawl behavior beyond Search Console?](https://www.theseocentral.com/technical-seo-faq/how-do-you-analyze-googlebot-crawl-behavior-beyond-search-console.md)
- [How do you decide which URL patterns should be crawlable?](https://www.theseocentral.com/technical-seo-faq/how-do-you-decide-which-url-patterns-should-be-crawlable.md)
- [How do you determine whether a website actually has a crawl-budget problem?](https://www.theseocentral.com/technical-seo-faq/how-do-you-determine-whether-a-website-actually-has-a-crawl-budget-problem.md)
- [How do you differentiate crawl demand from crawl capacity?](https://www.theseocentral.com/technical-seo-faq/how-do-you-differentiate-crawl-demand-from-crawl-capacity.md)
- [How do you identify URLs that unnecessarily consume crawler resources?](https://www.theseocentral.com/technical-seo-faq/how-do-you-identify-urls-that-unnecessarily-consume-crawler-resources.md)
- [How would you handle millions of faceted-navigation URLs?](https://www.theseocentral.com/technical-seo-faq/how-would-you-handle-millions-of-faceted-navigation-urls.md)
- [How would you optimize crawling on a website with millions of URLs?](https://www.theseocentral.com/technical-seo-faq/how-would-you-optimize-crawling-on-a-website-with-millions-of-urls.md)
- [When would you use robots.txt versus noindex?](https://www.theseocentral.com/technical-seo-faq/when-would-you-use-robots-txt-versus-noindex.md)

---

Answered by Abhinav Krishna (https://www.theseocentral.com/author/abhinav-krishna)
Source: https://www.theseocentral.com/technical-seo-faq/what-patterns-in-server-logs-indicate-inefficient-crawling