Technical SEO FAQ · XML Sitemaps & Robots.txt
What are common mistakes in robots.txt?
Common mistakes include:
- accidentally blocking important directories;
- blocking CSS/JS needed for rendering;
- using robots.txt as a substitute for noindex;
- incorrect path matching;
- forgetting protocol/host scope;
- assuming robots.txt removes already-known URLs;
- testing only one URL when a wildcard affects thousands.
Google's robots.txt rules apply to the relevant host, protocol and port.
Related questions
- How do sitemap errors help diagnose technical SEO problems?
- How do you design XML sitemaps for millions of URLs?
- How do you structure sitemap indexes for large websites?
- How do you troubleshoot sitemap URLs that Google isn't indexing?
- How would you safely change robots.txt on a high-traffic production website?
- Should canonicalized URLs appear in XML sitemaps?
- Should noindexed URLs appear in XML sitemaps?
- What happens if a URL is blocked by robots.txt but included in an XML sitemap?
- What should and shouldn't be included in an XML sitemap?
Community answers
No community answers yet.
Log in to answer Members can add their own answer and like others.