Robots.txt Analyzer and Tester
Read the robots.txt a site is serving right now and find out what its rules really do to Googlebot and other crawlers, line by line.
What the analyzer tests in your robots.txt
Rules are not matched as text. Each check asks what the file actually decides for a sample path, using the same group selection and longest-match logic crawlers follow.
- Whole-site blocks
- Whether the group for all crawlers, or the rules Googlebot follows, disallow the entire site.
- Scripts and stylesheets
- Sample CSS and JavaScript paths, including WordPress theme and includes files, tested against Googlebot's rules.
- Server errors
- A robots.txt that answers with a 5xx status is flagged as critical, because Google then treats the whole site as off limits.
- Syntax
- Lines that are not valid directives, or rules sitting outside any User-agent group, listed by line number.
- File size
- Whether the file fits inside the 500 KiB Google reads. Anything past that limit is ignored.
- Sitemap lines
- Whether a sitemap is declared at all, and whether each Sitemap line is a full absolute URL.
- WordPress and site paths
- What happens to /wp-admin/, /wp-includes/, /wp-json/, ?rest_route=, feeds, internal search and URLs with parameters.
- Crawl-delay, Host and groups
- Which groups set a Crawl-delay, whether a Host line exists, and how many User-agent groups the file has.
- Ten named crawlers
- Whether archive, SEO and data crawlers such as ia_archiver, CCBot, Bytespider and PetalBot have rules of their own or are blocked.
- WooCommerce (optional)
- Cart, checkout, my-account and add-to-cart URLs, plus whether a product sitemap is declared.
How to analyze a robots.txt file
Enter the website
Any page address works. The tool reads /robots.txt at the root of that host, so test www and non-www separately if both serve content.
Tick WooCommerce if it is a shop
This adds the five store checks for cart, checkout, account, add-to-cart links and a product sitemap.
Run it and read the failures first
Failed checks decide whether the site can be crawled at all. Warnings come next, then informational notes.
Edit, publish, run again
Change the lines the evidence points to, upload the new file and repeat the test to confirm.
The file is fetched with an 8 second limit and up to 600 KiB is read. If the file is larger, only the beginning is analysed and the result says so.
Understanding the analyzer's findings
- Passed, Warnings, Failed, Total Checks
- The overall count. A single failure here usually matters more than every warning together.
- Evidence
- The exact rule that decided the answer, with its line number, so you can find it in the file at once.
- "No rules of its own"
- That crawler has no group naming it and follows the rules for all crawlers.
- Blocked vs not blocked on WordPress paths
- These are choices, not errors. Blocking /wp-admin/ is common. Blocking /wp-includes/ can hide files Google needs to render pages.
- No robots.txt found
- The address answered 4xx or returned a normal page. Crawlers then assume everything may be crawled.
- Raw file
- The robots.txt text as it was served, next to the parsed groups and declared sitemaps.
Robots.txt mistakes it catches, and the fix
Disallow: / left in the User-agent: * group after launch
Fix: Delete that line or replace it with the specific paths you want hidden. A staging rule that survives launch keeps the whole site out of search.
CSS or JavaScript blocked for Googlebot
Fix: Remove the Disallow lines that cover theme, asset or include folders, or add Allow lines for .css and .js files inside them.
robots.txt returns a server error
Fix: Make /robots.txt answer 200 with your rules, or 404 if you have none. Check that a firewall or broken rewrite is not returning 500 or 503.
Sitemap written as a relative path
Fix: Write the full address, for example Sitemap: https://example.com/sitemap.xml. Crawlers ignore relative Sitemap lines.
Typos and lines outside a group
Fix: Check the listed line numbers. Misspelled directives and rules before the first User-agent line are skipped by crawlers.
Shop cart and add-to-cart URLs open to crawlers
Fix: Add Disallow lines for /cart/, /checkout/, /my-account/ and /*?add-to-cart= so crawl time goes to products and categories.
Who should run a robots.txt check
- Anyone who just launched a site or moved it off a staging server.
- SEO auditors who need the deciding rule and line number for a client report.
- WordPress and WooCommerce owners whose plugins may have rewritten the file.
- Developers reviewing a robots.txt change before it goes live.
Blocked is not the same as removed
Disallow stops crawling, not indexing. A blocked URL that other sites link to can still appear in search with no description, because Google cannot read the page to see a noindex tag. To keep a page out of results, allow crawling and use noindex, or protect it with a password.
Robots.txt Analyzer FAQ
Short answers about the Robots.txt Analyzer.
28 on every file. Ticking the WooCommerce option adds 5 more for shop paths and the product sitemap.
That is allowed. A 404 tells crawlers everything may be crawled. The tool reports it as information and suggests adding one to point crawlers to your sitemap.
When Google cannot read robots.txt because of a server error, it treats the whole site as disallowed until the file is readable again. A file that is missing is harmless; one that errors is not.
No. Google ignores Crawl-delay. Bing and Yandex honour it, so the check lists which groups set one.
This tool tests the file as a whole with sample paths. To see whether particular pages are open to AI crawlers, use the AI Crawler Access Check, and for one page and Googlebot use Bot Viewer.
Each host has its own robots.txt. Run the analyzer once for each subdomain you care about, such as shop. or blog.
Related SEO tools
Robots.txt Generator
A clean robots.txt for WordPress, Shopify or any site.
Free, unlimitedAI Crawler Access Check
Which AI and search crawlers can reach your pages.
1 free run a dayBot Viewer
Compare what a visitor and Googlebot receive.
1 free run a dayXML Sitemap Extractor
Every URL inside a sitemap or sitemap index.
1 free run a day