AI crawler checker

Enter a website and see which AI crawlers its robots.txt allows or blocks: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more.

We fetch the robots.txt file once from our server and keep the result in memory for up to 10 minutes. We don’t store the address you enter.

The crawlers we check

AI crawler tokens
Crawler Operator Purpose User agent token
GPTBot OpenAI AI training crawler GPTBot
OAI-SearchBot OpenAI AI search crawler OAI-SearchBot
ChatGPT-User OpenAI Fetches a page when a person asks an assistant ChatGPT-User
ClaudeBot Anthropic AI training crawler ClaudeBot
Claude-SearchBot Anthropic AI search crawler Claude-SearchBot
Claude-User Anthropic Fetches a page when a person asks an assistant Claude-User
Claude-Web Anthropic Fetches a page when a person asks an assistant Claude-Web
anthropic-ai Anthropic AI training crawler anthropic-ai
PerplexityBot Perplexity AI search crawler PerplexityBot
Perplexity-User Perplexity Fetches a page when a person asks an assistant Perplexity-User
Google-Extended Google AI training crawler robots.txt token only
Google-CloudVertexBot Google Fetches a page when a person asks an assistant Google-CloudVertexBot
GoogleOther Google AI training crawler GoogleOther
Applebot-Extended Apple AI training crawler robots.txt token only
Bytespider ByteDance AI training crawler Bytespider
CCBot Common Crawl AI training crawler CCBot
Meta-ExternalAgent Meta AI training crawler meta-externalagent
Meta-ExternalFetcher Meta Fetches a page when a person asks an assistant meta-externalfetcher
Amazonbot Amazon AI search crawler Amazonbot
DuckAssistBot DuckDuckGo Fetches a page when a person asks an assistant DuckAssistBot
MistralAI-User Mistral AI Fetches a page when a person asks an assistant MistralAI-User
cohere-ai Cohere Fetches a page when a person asks an assistant cohere-ai
cohere-training-data-crawler Cohere AI training crawler cohere-training-data-crawler
YouBot You.com AI search crawler YouBot
AI2Bot Allen Institute for AI AI training crawler AI2Bot
Diffbot Diffbot AI training crawler Diffbot

What robots.txt can’t do

robots.txt is a request, not a lock: a crawler can ignore it. And it doesn’t tell you which crawlers actually visit. That’s what the Wholetally AI report shows, from your own traffic.

See the AI report

Count the visitors you’ve been missing.

30-day free trial, no card needed. Create an account with your email and add your first site.

Start 30-day trial