Robots.txt Generator

Runs in browser

Generate robots.txt files visually with AI scraper blocking and sitemap URLs.

Generate robots.txt files visually with a rule builder. Control which search engines can crawl your site, block AI scrapers, set crawl delays, and add sitemap URLs.

Robots.txt Generator tool

robots.txt preview
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

🔒 Runs in your browser · No uploads · Your data never leaves your device

How to use

  1. Choose a preset

    Start with Allow all, Block all, or Block AI scrapers for common configurations.

  2. Add rules

    Add Allow and Disallow path rules for each User-agent group. Use * for all bots.

  3. Download

    Copy or download robots.txt, then upload it to your website root.

Common use cases

  • Blocking AI training crawlers — Toggle 'Block AI scrapers' to add Disallow rules for GPTBot, CCBot, anthropic-ai, and other known AI crawlers.
  • Protecting private areas from Google — Add Disallow rules for /admin/, /staging/, or other internal paths to prevent them from being indexed.

Examples

  • Block admin paths

    Allow crawling but block /admin/ and /private/.

    Output
    User-agent: * / Disallow: /admin/ / Disallow: /private/ / Sitemap: https://example.com/sitemap.xml

Frequently asked questions

Where do I put robots.txt?
robots.txt must be at the root of your domain: https://yourdomain.com/robots.txt
Does blocking AI scrapers in robots.txt actually work?
Reputable crawlers like GPTBot honour robots.txt. However, low-quality scrapers may ignore it. robots.txt is an advisory protocol, not a technical barrier.

Key concepts

User-agent
The identifier for a web crawler in robots.txt. Use * to match all bots, or a specific name like Googlebot to target one crawler.
Disallow
A robots.txt directive that tells crawlers not to request a specific path or directory.

Related tools

You might find these useful too.

More network tools