About the Robots.txt Generator
robots.txt tells search engine crawlers which parts of your site they may crawl. Create rule groups for all bots or specific ones (Googlebot, Bingbot), add your sitemap URL and optionally block common AI training crawlers such as GPTBot and CCBot.
How to use it
- Add rule groups with allowed and disallowed paths.
- Enter your sitemap URL.
- Download robots.txt and upload it to your site's root.
Good to know
- robots.txt does not hide pages: blocked URLs can still appear in search results if linked elsewhere. Use noindex for that.
- Disallow: (empty) means allow everything.