Technical SEO FAQ · XML Sitemaps & Robots.txt
How do sitemap errors help diagnose technical SEO problems?
A sitemap can function as a controlled test set.
If your sitemap contains 100,000 URLs and only 50,000 are indexed, investigate:
- duplicates;
- canonical mismatches;
- noindex;
- content quality;
- discovery;
- URL status;
- rendering.
Do not interpret the indexed/submitted ratio as a universal "indexation score." The reasons vary by site.
Related questions
- How do you design XML sitemaps for millions of URLs?
- How do you structure sitemap indexes for large websites?
- How do you troubleshoot sitemap URLs that Google isn't indexing?
- How would you safely change robots.txt on a high-traffic production website?
- Should canonicalized URLs appear in XML sitemaps?
- Should noindexed URLs appear in XML sitemaps?
- What are common mistakes in robots.txt?
- What happens if a URL is blocked by robots.txt but included in an XML sitemap?
- What should and shouldn't be included in an XML sitemap?
