Skip to content

Robots.txt Generator

Start from a sensible template for your platform, add your sitemap and any rules of your own, and get a file where every line actually does something.

Loading the tool

What your new robots.txt can contain

The generator writes the file in a fixed order: the platform template, the AI crawler block if you chose it, your own rules, then the Sitemap line.

Any site template
A single group that allows everything, the right starting point when you have nothing to hide.
WordPress template
Blocks /wp-admin/ while keeping admin-ajax.php open, plus xmlrpc.php, readme.html and internal search URLs. /wp-includes/ stays open so Google can load the scripts and styles your pages need.
WordPress and WooCommerce template
The WordPress basics plus cart, checkout, my-account, add-to-cart links and sorted product lists (?orderby=).
Shopify template
Admin, cart, checkout, orders, search and filtered collection URLs that combine tags with a plus sign.
AI crawler block
A Disallow: / group for each of 16 tokens: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, Bytespider, Applebot-Extended, Meta-ExternalAgent, CCBot, cohere-ai, Omgilibot and FacebookBot.
Your own rules, validated
Only real directives are kept: User-agent, Allow, Disallow, Sitemap, Crawl-delay, Host and Clean-param. Comments starting with # stay as written.
A checked Sitemap line
Added only when the address is a full http or https URL, because crawlers skip relative ones.

How to create a robots.txt file

  1. Choose your platform

    Any site, WordPress, WordPress with WooCommerce, or Shopify. The template covers the paths that platform usually should not have crawled.

  2. Add your sitemap address

    Paste the full URL of your sitemap or sitemap index. Leave it empty if you do not have one yet.

  3. Decide on AI crawlers

    Tick the box to block the 16 AI and LLM crawlers. Leave it off if you want AI assistants to read and cite your pages.

  4. Type extra rules if needed

    For example a User-agent line for one bot followed by its Disallow lines. Anything a crawler would not honour is dropped and explained.

  5. Copy or download and upload

    Save the result as robots.txt in the root of your domain, where crawlers look for it: https://yourdomain/robots.txt.

Custom rules can be up to 4,000 characters. Nothing is fetched from your site: the file is built only from what you enter.

Reading the generator's output

Groups and Rules
How many User-agent groups the file has and how many Allow or Disallow lines sit in them.
AI Crawlers Blocked
The number of AI tokens that received a Disallow: / group. Zero if you left the box unticked.
File Size
The size of the finished file in KiB.
Notes
Every line that was dropped, with the reason, plus warnings such as a missing sitemap or a whole-site block.
The robots.txt preview
The exact text to publish, ready to copy or download as a file.

Mistakes the generator stops for you

  • A rule typed before any User-agent line

    Fix: It is dropped, because no crawler applies a rule outside a group. Put a User-agent line above it and generate again.

  • A path written without a leading slash

    Fix: Lines such as Disallow: private/ are dropped. Write Disallow: /private/ so the rule matches from the root.

  • A made-up or misspelled directive

    Fix: Lines like Noindex: or Dissallow: are removed with a note. Google stopped supporting noindex in robots.txt, so use a meta robots tag on the page instead.

  • Disallow: / for every crawler

    Fix: The result opens with a warning that this asks Google to stop crawling the whole site. Publish it only for a site that must stay out of search.

  • A relative sitemap address

    Fix: The Sitemap line is left out with a note. Enter the complete address, such as https://example.com/sitemap.xml.

Who the robots.txt generator helps

  • New site owners who need a correct file on day one without learning the syntax.
  • WordPress and WooCommerce stores that want admin, cart and account pages kept out of crawls.
  • Shopify merchants checking which paths a sensible file covers.
  • Publishers deciding whether AI crawlers may use their writing.

Blocking AI crawlers is a trade-off

The block list includes training crawlers and also the bots behind AI search and on-demand fetching, such as OAI-SearchBot, PerplexityBot and ChatGPT-User. Blocking them keeps your pages out of model training, but it can also keep you out of answers in those assistants. Googlebot is never in the list, so Google Search and your Google rankings are not touched.

Check theme files after publishing

The WordPress templates leave /wp-includes/ and your theme folders open, because Google needs the CSS and JavaScript there to render your pages. After you upload the file, run the Robots.txt Analyzer to confirm Googlebot can still fetch everything your pages need.

Robots.txt Generator FAQ

Short answers about the Robots.txt Generator.

In the root folder of your domain, so it loads at https://yourdomain.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own.