Glossary
What Is Robots.txt?
Robots.txt is a plain-text file placed at a website's root directory that tells search engine crawlers which URL paths they may or may not request and crawl.
Robots.txt controls crawling, not indexing — a disallowed page can still be indexed (usually without a description) if other pages link to it, which is a frequent point of confusion.
Want to go beyond definitions and actually fix issues like this on your own site? Join the Technical SEO Workshop →