AI crawlers: who reads your site, and why
The AI user agents we know about, what each one is for, and whether it can be seen in your traffic.
| Crawler | Operator | Purpose | User agent token |
|---|---|---|---|
| GPTBot | OpenAI | AI training crawler | GPTBot |
| OAI-SearchBot | OpenAI | AI search crawler | OAI-SearchBot |
| ChatGPT-User | OpenAI | Fetches a page when a person asks an assistant | ChatGPT-User |
| ClaudeBot | Anthropic | AI training crawler | ClaudeBot |
| Claude-SearchBot | Anthropic | AI search crawler | Claude-SearchBot |
| Claude-User | Anthropic | Fetches a page when a person asks an assistant | Claude-User |
| Claude-Web | Anthropic | Fetches a page when a person asks an assistant | Claude-Web |
| anthropic-ai | Anthropic | AI training crawler | anthropic-ai |
| PerplexityBot | Perplexity | AI search crawler | PerplexityBot |
| Perplexity-User | Perplexity | Fetches a page when a person asks an assistant | Perplexity-User |
| Google-Extended | AI training crawler | robots.txt token only | |
| Google-CloudVertexBot | Fetches a page when a person asks an assistant | Google-CloudVertexBot | |
| GoogleOther | AI training crawler | GoogleOther | |
| Applebot-Extended | Apple | AI training crawler | robots.txt token only |
| Bytespider | ByteDance | AI training crawler | Bytespider |
| CCBot | Common Crawl | AI training crawler | CCBot |
| Meta-ExternalAgent | Meta | AI training crawler | meta-externalagent |
| Meta-ExternalFetcher | Meta | Fetches a page when a person asks an assistant | meta-externalfetcher |
| Amazonbot | Amazon | AI search crawler | Amazonbot |
| DuckAssistBot | DuckDuckGo | Fetches a page when a person asks an assistant | DuckAssistBot |
| MistralAI-User | Mistral AI | Fetches a page when a person asks an assistant | MistralAI-User |
| cohere-ai | Cohere | Fetches a page when a person asks an assistant | cohere-ai |
| cohere-training-data-crawler | Cohere | AI training crawler | cohere-training-data-crawler |
| YouBot | You.com | AI search crawler | YouBot |
| AI2Bot | Allen Institute for AI | AI training crawler | AI2Bot |
| Diffbot | Diffbot | AI training crawler | Diffbot |
This is the list the Wholetally edge collector recognises. We re-check it against the operators’ documentation from time to time.
Count the visitors you’ve been missing.
30-day free trial, no card needed. Create an account with your email and add your first site.