What Is a robots.txt File?
A robots.txt file is a plain text file placed at the root of a website that tells search engine crawlers which pages or sections they are allowed or not allowed to access. It follows the Robots Exclusion Protocol, an industry-standard mechanism for controlling bot behavior. Properly configuring robots.txt is essential for SEO because it guides crawlers toward valuable content and away from duplicate, staging, or private areas. A misconfigured file can accidentally block important pages from indexing or expose sensitive directories.
How to Use This Generator
- Select the user-agent (or keep
*to target all bots). - Add paths you want to allow or disallow for the selected bot.
- Optionally enter your sitemap URL and a crawl-delay value.
- Click "Generate robots.txt" to produce the file content.
- Copy the output and save it as
robots.txtin your site's root directory.
Frequently Asked Questions
Where should I place the robots.txt file?
It must be at the root of your domain, e.g., https://example.com/robots.txt. Placing it in a subdirectory will have no effect on crawler behavior.
Does robots.txt guarantee pages won't be indexed?
No. While most major search engines respect robots.txt, it is a directive, not a guarantee. For stronger protection, use the noindex meta tag or HTTP header.