TheSEOCentral.

Technical SEO FAQ · XML Sitemaps & Robots.txt

What are common mistakes in robots.txt?

Common mistakes include:

  • accidentally blocking important directories;
  • blocking CSS/JS needed for rendering;
  • using robots.txt as a substitute for noindex;
  • incorrect path matching;
  • forgetting protocol/host scope;
  • assuming robots.txt removes already-known URLs;
  • testing only one URL when a wildcard affects thousands.

Google's robots.txt rules apply to the relevant host, protocol and port.

Sources

Related questions

← All technical SEO questions View as Markdown (.md)