Technical SEO FAQ · Crawling, Crawl Budget & URL Management
How would you handle millions of faceted-navigation URLs?
First determine which facet combinations have genuine search value.
Separate:
SEO landing pages
from
UI filtering states.
For example:
/shoes/nike/
may be an intentional landing page.
Whereas:
/shoes?color=red&sort=price&session=123
may simply be a user interface state.
For valuable combinations, create deliberate, indexable URLs with unique content and clear internal linking.
For unnecessary combinations, consolidate or prevent crawling where appropriate.
Google's current crawling documentation specifically addresses faceted navigation and warns that unbounded combinations can create huge URL spaces.
Related questions
- Can blocking a URL in robots.txt solve an indexation problem?
- How do you analyze Googlebot crawl behavior beyond Search Console?
- How do you decide which URL patterns should be crawlable?
- How do you determine whether a website actually has a crawl-budget problem?
- How do you differentiate crawl demand from crawl capacity?
- How do you identify URLs that unnecessarily consume crawler resources?
- How would you optimize crawling on a website with millions of URLs?
- What patterns in server logs indicate inefficient crawling?
- When would you use robots.txt versus noindex?