AI crawler checker
Enter a website and see which AI crawlers its robots.txt allows or blocks: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more.
We fetch the robots.txt file once from our server and keep the result in memory for up to 10 minutes. We don’t store the address you enter.
The crawlers we check
| Crawler | Operator | Purpose | User agent token |
|---|---|---|---|
| GPTBot | OpenAI | AI training crawler | GPTBot |
| OAI-SearchBot | OpenAI | AI search crawler | OAI-SearchBot |
| ChatGPT-User | OpenAI | Fetches a page when a person asks an assistant | ChatGPT-User |
| ClaudeBot | Anthropic | AI training crawler | ClaudeBot |
| Claude-SearchBot | Anthropic | AI search crawler | Claude-SearchBot |
| Claude-User | Anthropic | Fetches a page when a person asks an assistant | Claude-User |
| Claude-Web | Anthropic | Fetches a page when a person asks an assistant | Claude-Web |
| anthropic-ai | Anthropic | AI training crawler | anthropic-ai |
| PerplexityBot | Perplexity | AI search crawler | PerplexityBot |
| Perplexity-User | Perplexity | Fetches a page when a person asks an assistant | Perplexity-User |
| Google-Extended | AI training crawler | robots.txt token only | |
| Google-CloudVertexBot | Fetches a page when a person asks an assistant | Google-CloudVertexBot | |
| GoogleOther | AI training crawler | GoogleOther | |
| Applebot-Extended | Apple | AI training crawler | robots.txt token only |
| Bytespider | ByteDance | AI training crawler | Bytespider |
| CCBot | Common Crawl | AI training crawler | CCBot |
| Meta-ExternalAgent | Meta | AI training crawler | meta-externalagent |
| Meta-ExternalFetcher | Meta | Fetches a page when a person asks an assistant | meta-externalfetcher |
| Amazonbot | Amazon | AI search crawler | Amazonbot |
| DuckAssistBot | DuckDuckGo | Fetches a page when a person asks an assistant | DuckAssistBot |
| MistralAI-User | Mistral AI | Fetches a page when a person asks an assistant | MistralAI-User |
| cohere-ai | Cohere | Fetches a page when a person asks an assistant | cohere-ai |
| cohere-training-data-crawler | Cohere | AI training crawler | cohere-training-data-crawler |
| YouBot | You.com | AI search crawler | YouBot |
| AI2Bot | Allen Institute for AI | AI training crawler | AI2Bot |
| Diffbot | Diffbot | AI training crawler | Diffbot |
What robots.txt can’t do
robots.txt is a request, not a lock: a crawler can ignore it. And it doesn’t tell you which crawlers actually visit. That’s what the Wholetally AI report shows, from your own traffic.
Count the visitors you’ve been missing.
30-day free trial, no card needed. Create an account with your email and add your first site.