How do I block AI crawlers?

Select bots (e.g., GPTBot, ClaudeBot) and generate Disallow rules; upload to /robots.txt and validate.

Is robots.txt enforceable?

Most reputable crawlers honor robots.txt. Some rogue bots may ignore it—monitor and block at the edge if needed.

Free robots.txt Generator & Validator

Free: create and validate robots.txt. Choose bots, allow/disallow, add crawl-delay and sitemap, copy or download.

Choose bots

Checked = Disallow. Unchecked = Allow.

Disallow specific paths (optional)

Crawl-delay (seconds, optional)Sitemap URL (optional)Comments (not served to bots)

Preview

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: CCBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Bytespider
User-agent: Amazonbot
Allow: /