Skip to content

Robots.txt Analyzer and Tester

Read the robots.txt a site is serving right now and find out what its rules really do to Googlebot and other crawlers, line by line.

Loading the tool

What the analyzer tests in your robots.txt

Rules are not matched as text. Each check asks what the file actually decides for a sample path, using the same group selection and longest-match logic crawlers follow.

Whole-site blocks
Whether the group for all crawlers, or the rules Googlebot follows, disallow the entire site.
Scripts and stylesheets
Sample CSS and JavaScript paths, including WordPress theme and includes files, tested against Googlebot's rules.
Server errors
A robots.txt that answers with a 5xx status is flagged as critical, because Google then treats the whole site as off limits.
Syntax
Lines that are not valid directives, or rules sitting outside any User-agent group, listed by line number.
File size
Whether the file fits inside the 500 KiB Google reads. Anything past that limit is ignored.
Sitemap lines
Whether a sitemap is declared at all, and whether each Sitemap line is a full absolute URL.
WordPress and site paths
What happens to /wp-admin/, /wp-includes/, /wp-json/, ?rest_route=, feeds, internal search and URLs with parameters.
Crawl-delay, Host and groups
Which groups set a Crawl-delay, whether a Host line exists, and how many User-agent groups the file has.
Ten named crawlers
Whether archive, SEO and data crawlers such as ia_archiver, CCBot, Bytespider and PetalBot have rules of their own or are blocked.
WooCommerce (optional)
Cart, checkout, my-account and add-to-cart URLs, plus whether a product sitemap is declared.

How to analyze a robots.txt file

  1. Enter the website

    Any page address works. The tool reads /robots.txt at the root of that host, so test www and non-www separately if both serve content.

  2. Tick WooCommerce if it is a shop

    This adds the five store checks for cart, checkout, account, add-to-cart links and a product sitemap.

  3. Run it and read the failures first

    Failed checks decide whether the site can be crawled at all. Warnings come next, then informational notes.

  4. Edit, publish, run again

    Change the lines the evidence points to, upload the new file and repeat the test to confirm.

The file is fetched with an 8 second limit and up to 600 KiB is read. If the file is larger, only the beginning is analysed and the result says so.

Understanding the analyzer's findings

Passed, Warnings, Failed, Total Checks
The overall count. A single failure here usually matters more than every warning together.
Evidence
The exact rule that decided the answer, with its line number, so you can find it in the file at once.
"No rules of its own"
That crawler has no group naming it and follows the rules for all crawlers.
Blocked vs not blocked on WordPress paths
These are choices, not errors. Blocking /wp-admin/ is common. Blocking /wp-includes/ can hide files Google needs to render pages.
No robots.txt found
The address answered 4xx or returned a normal page. Crawlers then assume everything may be crawled.
Raw file
The robots.txt text as it was served, next to the parsed groups and declared sitemaps.

Robots.txt mistakes it catches, and the fix

  • Disallow: / left in the User-agent: * group after launch

    Fix: Delete that line or replace it with the specific paths you want hidden. A staging rule that survives launch keeps the whole site out of search.

  • CSS or JavaScript blocked for Googlebot

    Fix: Remove the Disallow lines that cover theme, asset or include folders, or add Allow lines for .css and .js files inside them.

  • robots.txt returns a server error

    Fix: Make /robots.txt answer 200 with your rules, or 404 if you have none. Check that a firewall or broken rewrite is not returning 500 or 503.

  • Sitemap written as a relative path

    Fix: Write the full address, for example Sitemap: https://example.com/sitemap.xml. Crawlers ignore relative Sitemap lines.

  • Typos and lines outside a group

    Fix: Check the listed line numbers. Misspelled directives and rules before the first User-agent line are skipped by crawlers.

  • Shop cart and add-to-cart URLs open to crawlers

    Fix: Add Disallow lines for /cart/, /checkout/, /my-account/ and /*?add-to-cart= so crawl time goes to products and categories.

Who should run a robots.txt check

  • Anyone who just launched a site or moved it off a staging server.
  • SEO auditors who need the deciding rule and line number for a client report.
  • WordPress and WooCommerce owners whose plugins may have rewritten the file.
  • Developers reviewing a robots.txt change before it goes live.

Blocked is not the same as removed

Disallow stops crawling, not indexing. A blocked URL that other sites link to can still appear in search with no description, because Google cannot read the page to see a noindex tag. To keep a page out of results, allow crawling and use noindex, or protect it with a password.

Robots.txt Analyzer FAQ

Short answers about the Robots.txt Analyzer.

28 on every file. Ticking the WooCommerce option adds 5 more for shop paths and the product sitemap.