Skip to content

AI Crawler Access Checker

Find out which AI assistants and AI search engines may read your pages, page by page, and the exact robots.txt line that allows or blocks each one.

Loading the tool

Which AI crawlers are tested

Every page you enter is tested against all 27 tokens. Each crawler carries its vendor and its purpose, so you can tell a training crawler from one that brings visitors.

OpenAI
GPTBot for training, OAI-SearchBot for ChatGPT search results, and ChatGPT-User for pages a user asks ChatGPT to open.
Anthropic
ClaudeBot for training, Claude-SearchBot for Claude's web search, Claude-User for on-demand fetches, and the older Claude-Web and anthropic-ai tokens.
Google
Google-Extended for Gemini training and grounding, Googlebot (AI Overviews are built from the Search index), and GoogleOther for research.
Perplexity and Apple
PerplexityBot and Perplexity-User, plus Applebot for Siri and Spotlight and Applebot-Extended for Apple's model training.
Meta, Amazon and ByteDance
Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot, Amazonbot and Bytespider.
Datasets and smaller assistants
CCBot (Common Crawl), cohere-ai, cohere-training-data-crawler, MistralAI-User, YouBot, DuckAssistBot and Diffbot.
Per page, not per site
Rules are matched against each URL's own path and query, so a Disallow on /blog/ shows up only on blog pages.

How to check AI crawler access

  1. List the pages

    Paste up to 25 addresses, one per line, from as many as 10 different sites. Mix your key pages with a competitor's if you want a comparison.

  2. Run the check

    Each site's robots.txt is fetched once, then every page is tested against every crawler.

  3. Scan the summary

    See how many crawlers reach the site and how many are fully blocked across the pages you entered.

  4. Open a page row

    For each crawler: allowed, blocked or unknown, whether a group names it or it follows the * group, and the rule with its line number.

Duplicate addresses are removed. Each robots.txt fetch has an 8 second limit.

Reading the access results

URLs Analyzed and AI Crawlers Tested
How many pages were checked and against how many tokens.
Crawlers Reaching Site
Tokens allowed on at least part of what you tested.
Fully Blocked
Tokens that cannot fetch any of the pages you entered.
Purpose: training, search, on-demand, legacy, research
Training crawlers collect text for models. Search and on-demand bots fetch pages to answer a user and can send visits. Legacy tokens are old names still seen in files.
Named group vs * group
A crawler with its own group ignores the * rules entirely. This is the most common reason a block does not work as intended.
Unknown
robots.txt answered with a server error or could not be reached, so the answer cannot be known. Crawlers following the standard treat that as blocked.

AI crawler access problems, and the fix

  • You blocked GPTBot but ChatGPT still cannot cite you, or still can

    Fix: GPTBot is only training. OAI-SearchBot decides ChatGPT search and ChatGPT-User handles user requests. Set a rule for each token according to what you want.

  • A named group that accidentally opens everything

    Fix: If User-agent: ClaudeBot has only an Allow line, ClaudeBot skips your * rules, including private folders. Copy the Disallow lines you need into its group.

  • You want out of Gemini training but in Google Search

    Fix: Disallow Google-Extended and leave Googlebot allowed. Google-Extended does not affect Search rankings or crawling.

  • Results show unknown for a whole site

    Fix: Fix the server error or firewall rule on /robots.txt so it answers 200 or 404. Until then, standard crawlers stay away from the whole site.

  • Your pages never appear in Perplexity answers

    Fix: Check whether PerplexityBot is blocked by the * group or a security rule. Allow it if you want to be cited.

Who checks AI crawler access

  • Publishers deciding which AI companies may train on their articles.
  • Brands that want to be cited in ChatGPT, Claude, Perplexity and Google AI answers.
  • SEO and AEO specialists auditing a client's visibility in AI search.
  • Developers confirming that a new robots.txt does what legal or editorial asked for.

robots.txt is a request, not a wall

This tool shows what each crawler is told. Reputable AI companies say their bots obey robots.txt, but the file cannot force anything. If a firewall or bot protection also blocks AI user agents, the real access can be stricter than the file suggests.

AI Crawler Access Check FAQ

Short answers about the AI Crawler Access Check.

Up to 25 page addresses per run, spread over at most 10 different sites.