Technical SEO FAQ · XML Sitemaps & Robots.txt
What happens if a URL is blocked by robots.txt but included in an XML sitemap?
The sitemap tells Google the URL exists, while robots.txt tells Google it cannot crawl the URL.
This creates conflicting operational signals.
Google can know the URL exists even though it cannot fetch its content.
Do not use the sitemap to force Google to crawl a URL that robots.txt blocks.
Related questions
- How do sitemap errors help diagnose technical SEO problems?
- How do you design XML sitemaps for millions of URLs?
- How do you structure sitemap indexes for large websites?
- How do you troubleshoot sitemap URLs that Google isn't indexing?
- How would you safely change robots.txt on a high-traffic production website?
- Should canonicalized URLs appear in XML sitemaps?
- Should noindexed URLs appear in XML sitemaps?
- What are common mistakes in robots.txt?
- What should and shouldn't be included in an XML sitemap?
Community answers
No community answers yet.
Log in to answer Members can add their own answer and like others.