Technical SEO FAQ · Crawling, Crawl Budget & URL Management
When would you use robots.txt versus noindex?
Use robots.txt primarily for crawl control.
Use noindex for indexing control.
Example robots.txt:
User-agent: *
Disallow: /internal-search/
Example noindex:
<meta name="robots" content="noindex">
If Google must see a noindex directive, it must be able to crawl the page.
For pages you never want Google to crawl, robots.txt can be appropriate. For pages that need to be crawled so Google can see an indexing directive, noindex is appropriate.
Google's 2026 crawl-budget guidance also warns against using noindex merely as a crawl-budget tool: Google still has to request the URL to see the directive.
Related questions
- Can blocking a URL in robots.txt solve an indexation problem?
- How do you analyze Googlebot crawl behavior beyond Search Console?
- How do you decide which URL patterns should be crawlable?
- How do you determine whether a website actually has a crawl-budget problem?
- How do you differentiate crawl demand from crawl capacity?
- How do you identify URLs that unnecessarily consume crawler resources?
- How would you handle millions of faceted-navigation URLs?
- How would you optimize crawling on a website with millions of URLs?
- What patterns in server logs indicate inefficient crawling?