How do you identify URLs that unnecessarily consume crawler resources?

Look for:

• parameter URLs;
• duplicate URLs;
• unnecessary sorting/filtering combinations;
• internal search URLs;
• session URLs;
• redirect chains;
• soft 404s;
• infinite URL spaces;
• obsolete URLs;
• dynamically generated low-value URLs.

Use server logs to identify how often Googlebot requests each pattern.

A useful analysis is:

crawl requests by URL type ÷ total Googlebot requests

Then ask whether the percentage spent on each type is justified.

Google recommends managing URL inventory, consolidating duplicate content, and blocking unnecessary URLs where appropriate.