Technical SEO FAQ · XML Sitemaps & Robots.txt
How do you structure sitemap indexes for large websites?
Group URLs by logical types where useful:
Products
Categories
Editorial
Locations
Languages
This allows teams to detect which content system has a problem.
Also maintain accurate lastmod values when supported by your CMS workflow.
Google recommends using lastmod when it accurately reflects meaningful page changes.
Related questions
- How do sitemap errors help diagnose technical SEO problems?
- How do you design XML sitemaps for millions of URLs?
- How do you troubleshoot sitemap URLs that Google isn't indexing?
- How would you safely change robots.txt on a high-traffic production website?
- Should canonicalized URLs appear in XML sitemaps?
- Should noindexed URLs appear in XML sitemaps?
- What are common mistakes in robots.txt?
- What happens if a URL is blocked by robots.txt but included in an XML sitemap?
- What should and shouldn't be included in an XML sitemap?