AI Crawler Access Checker

Search, training and user retrieval are different

AI providers now publish separate crawlers for different jobs. This AI crawler checker keeps them apart so you can make a deliberate choice for each.

• Search and discovery: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude search results and PerplexityBot for Perplexity search.
• Model training: GPTBot at OpenAI and ClaudeBot at Anthropic. Blocking these is a separate decision from appearing in AI search, according to each provider.
• User-directed retrieval: ChatGPT-User, Claude-User and Perplexity-User fetch a page because a person asked. They are not automatic crawlers, and providers document that robots.txt may be treated differently.
• Standard search crawlers: Googlebot and Bingbot are search crawlers, not AI-only crawlers, although their indexes feed AI features in search.

How to use this GPTBot and OAI-SearchBot checker

1. Enter your domain. The tool fetches /robots.txt through a secure server request.
2. Optionally enter a path such as /services/seo to test a specific section.
3. Review the status and matched rule for each crawler, then copy a suggested rule or open the Robots.txt Generator.

How the rules are read

If a group names the crawler, only that group applies. Otherwise the wildcard group applies. Within a group the longest matching rule wins, and Allow wins a tie. This follows the robots.txt standard, but each provider can interpret edge cases differently, so treat results as a best-effort reading. You do not need to allow training crawlers to appear in AI search results. For help deciding, talk to our AI search team.