Technical SEO FAQ · XML Sitemaps & Robots.txt
How do you design XML sitemaps for millions of URLs?
Use sitemap indexes and split URL sets into manageable sitemap files according to the sitemap protocol limits.
Common segmentation:
sitemap-index.xml
├── products-001.xml
├── products-002.xml
├── categories.xml
├── articles-001.xml
└── locations.xml
Segmenting by content type also improves diagnostics.
Monitor each sitemap separately in Search Console.
Related questions
- How do sitemap errors help diagnose technical SEO problems?
- How do you structure sitemap indexes for large websites?
- How do you troubleshoot sitemap URLs that Google isn't indexing?
- How would you safely change robots.txt on a high-traffic production website?
- Should canonicalized URLs appear in XML sitemaps?
- Should noindexed URLs appear in XML sitemaps?
- What are common mistakes in robots.txt?
- What happens if a URL is blocked by robots.txt but included in an XML sitemap?
- What should and shouldn't be included in an XML sitemap?