# What are common mistakes in robots.txt?

> Common mistakes include:

- accidentally blocking important directories;

- blocking CSS/JS needed for rendering;

- using robots.txt as a substitute for noindex;

- incorrect path matching;

- forgetting protocol/host scope;

- assuming robots.txt removes already-known URLs;

- testing only one URL when a wildcard affects thousands.

Google's robots.txt rules apply to the relevant host, protocol and port.

## Sources

- [Google Search Central — Robots Txt Spec](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec)

## Related questions

- [How do sitemap errors help diagnose technical SEO problems?](https://www.theseocentral.com/technical-seo-faq/how-do-sitemap-errors-help-diagnose-technical-seo-problems.md)
- [How do you design XML sitemaps for millions of URLs?](https://www.theseocentral.com/technical-seo-faq/how-do-you-design-xml-sitemaps-for-millions-of-urls.md)
- [How do you structure sitemap indexes for large websites?](https://www.theseocentral.com/technical-seo-faq/how-do-you-structure-sitemap-indexes-for-large-websites.md)
- [How do you troubleshoot sitemap URLs that Google isn't indexing?](https://www.theseocentral.com/technical-seo-faq/how-do-you-troubleshoot-sitemap-urls-that-google-isn-t-indexing.md)
- [How would you safely change robots.txt on a high-traffic production website?](https://www.theseocentral.com/technical-seo-faq/how-would-you-safely-change-robots-txt-on-a-high-traffic-production-website.md)
- [Should canonicalized URLs appear in XML sitemaps?](https://www.theseocentral.com/technical-seo-faq/should-canonicalized-urls-appear-in-xml-sitemaps.md)
- [Should noindexed URLs appear in XML sitemaps?](https://www.theseocentral.com/technical-seo-faq/should-noindexed-urls-appear-in-xml-sitemaps.md)
- [What happens if a URL is blocked by robots.txt but included in an XML sitemap?](https://www.theseocentral.com/technical-seo-faq/what-happens-if-a-url-is-blocked-by-robots-txt-but-included-in-an-xml-sitemap.md)
- [What should and shouldn't be included in an XML sitemap?](https://www.theseocentral.com/technical-seo-faq/what-should-and-shouldn-t-be-included-in-an-xml-sitemap.md)

---

Answered by Abhinav Krishna (https://www.theseocentral.com/author/abhinav-krishna)
Source: https://www.theseocentral.com/technical-seo-faq/what-are-common-mistakes-in-robots-txt