Robots.txt Generator
Create a robots.txt file using a visual interface. Add rules for different user agents, allow and disallow patterns, and set your sitemap URL. Validates syntax and previews the output.
User-agent: * Allow: /api/public Disallow: /admin Disallow: /api/private User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml
Place this file at yourdomain.com/robots.txt — the root of your website.
Frequently Asked Questions
Should I block AI bots?
Your choice. Blocking GPTBot, Claude-Web, and CCBot prevents your content from being used to train AI models. However, AI search assistants (SearchGPT, Perplexity) also use these crawlers for real-time search — blocking them may reduce your visibility in AI search tools.
Does robots.txt guarantee bots won't access my pages?
No — robots.txt is advisory only. Well-behaved bots (Googlebot, Bingbot) respect it. Malicious crawlers ignore it. For actual access control, use server-side authentication. For preventing sensitive pages from appearing in search, use meta robots noindex tags instead.
What's the difference between Disallow and Allow?
Disallow blocks a path. Allow explicitly permits a path within a disallowed section. Example: Disallow /private allows you to block all of /private, then Allow /private/public-page permits just that one page. More specific rules take precedence.