AI Crawler Access Checker
Find out which AI assistants and AI search engines may read your pages, page by page, and the exact robots.txt line that allows or blocks each one.
Which AI crawlers are tested
Every page you enter is tested against all 27 tokens. Each crawler carries its vendor and its purpose, so you can tell a training crawler from one that brings visitors.
- OpenAI
- GPTBot for training, OAI-SearchBot for ChatGPT search results, and ChatGPT-User for pages a user asks ChatGPT to open.
- Anthropic
- ClaudeBot for training, Claude-SearchBot for Claude's web search, Claude-User for on-demand fetches, and the older Claude-Web and anthropic-ai tokens.
- Google-Extended for Gemini training and grounding, Googlebot (AI Overviews are built from the Search index), and GoogleOther for research.
- Perplexity and Apple
- PerplexityBot and Perplexity-User, plus Applebot for Siri and Spotlight and Applebot-Extended for Apple's model training.
- Meta, Amazon and ByteDance
- Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot, Amazonbot and Bytespider.
- Datasets and smaller assistants
- CCBot (Common Crawl), cohere-ai, cohere-training-data-crawler, MistralAI-User, YouBot, DuckAssistBot and Diffbot.
- Per page, not per site
- Rules are matched against each URL's own path and query, so a Disallow on /blog/ shows up only on blog pages.
How to check AI crawler access
List the pages
Paste up to 25 addresses, one per line, from as many as 10 different sites. Mix your key pages with a competitor's if you want a comparison.
Run the check
Each site's robots.txt is fetched once, then every page is tested against every crawler.
Scan the summary
See how many crawlers reach the site and how many are fully blocked across the pages you entered.
Open a page row
For each crawler: allowed, blocked or unknown, whether a group names it or it follows the * group, and the rule with its line number.
Duplicate addresses are removed. Each robots.txt fetch has an 8 second limit.
Reading the access results
- URLs Analyzed and AI Crawlers Tested
- How many pages were checked and against how many tokens.
- Crawlers Reaching Site
- Tokens allowed on at least part of what you tested.
- Fully Blocked
- Tokens that cannot fetch any of the pages you entered.
- Purpose: training, search, on-demand, legacy, research
- Training crawlers collect text for models. Search and on-demand bots fetch pages to answer a user and can send visits. Legacy tokens are old names still seen in files.
- Named group vs * group
- A crawler with its own group ignores the * rules entirely. This is the most common reason a block does not work as intended.
- Unknown
- robots.txt answered with a server error or could not be reached, so the answer cannot be known. Crawlers following the standard treat that as blocked.
AI crawler access problems, and the fix
You blocked GPTBot but ChatGPT still cannot cite you, or still can
Fix: GPTBot is only training. OAI-SearchBot decides ChatGPT search and ChatGPT-User handles user requests. Set a rule for each token according to what you want.
A named group that accidentally opens everything
Fix: If User-agent: ClaudeBot has only an Allow line, ClaudeBot skips your * rules, including private folders. Copy the Disallow lines you need into its group.
You want out of Gemini training but in Google Search
Fix: Disallow Google-Extended and leave Googlebot allowed. Google-Extended does not affect Search rankings or crawling.
Results show unknown for a whole site
Fix: Fix the server error or firewall rule on /robots.txt so it answers 200 or 404. Until then, standard crawlers stay away from the whole site.
Your pages never appear in Perplexity answers
Fix: Check whether PerplexityBot is blocked by the * group or a security rule. Allow it if you want to be cited.
Who checks AI crawler access
- Publishers deciding which AI companies may train on their articles.
- Brands that want to be cited in ChatGPT, Claude, Perplexity and Google AI answers.
- SEO and AEO specialists auditing a client's visibility in AI search.
- Developers confirming that a new robots.txt does what legal or editorial asked for.
robots.txt is a request, not a wall
This tool shows what each crawler is told. Reputable AI companies say their bots obey robots.txt, but the file cannot force anything. If a firewall or bot protection also blocks AI user agents, the real access can be stricter than the file suggests.
AI Crawler Access Check FAQ
Short answers about the AI Crawler Access Check.
Up to 25 page addresses per run, spread over at most 10 different sites.
GPTBot collects content to train OpenAI models. OAI-SearchBot indexes pages so they can be shown and linked in ChatGPT search. You can block one and allow the other.
No. AI Overviews are built from Google's Search index, which Googlebot crawls. Google-Extended controls use of your content for Gemini training and grounding only.
Every crawler is reported as allowed, because a missing file means no restrictions.
Somewhere in the file a group names that crawler. A crawler uses only the most specific group that matches it, so it ignores the * group. The deciding rule column shows which line won.
This tool only tests access. The Robots.txt Generator writes a file with a one-click block for 16 AI crawlers, and the LLMs.txt Generator helps AI assistants read the pages you do allow.
Related SEO tools
Robots.txt Generator
A clean robots.txt for WordPress, Shopify or any site.
Free, unlimitedRobots.txt Analyzer
28 checks on a live robots.txt file.
1 free run a dayLLMs.txt Generator
A draft llms.txt file built from your site.
1 free run a dayBot Viewer
Compare what a visitor and Googlebot receive.
1 free run a day