AI crawlers: who reads your site, and why

The AI user agents we know about, what each one is for, and whether it can be seen in your traffic.

AI crawlers and AI user agents
Crawler Operator Purpose User agent token
GPTBot OpenAI AI training crawler GPTBot
OAI-SearchBot OpenAI AI search crawler OAI-SearchBot
ChatGPT-User OpenAI Fetches a page when a person asks an assistant ChatGPT-User
ClaudeBot Anthropic AI training crawler ClaudeBot
Claude-SearchBot Anthropic AI search crawler Claude-SearchBot
Claude-User Anthropic Fetches a page when a person asks an assistant Claude-User
Claude-Web Anthropic Fetches a page when a person asks an assistant Claude-Web
anthropic-ai Anthropic AI training crawler anthropic-ai
PerplexityBot Perplexity AI search crawler PerplexityBot
Perplexity-User Perplexity Fetches a page when a person asks an assistant Perplexity-User
Google-Extended Google AI training crawler robots.txt token only
Google-CloudVertexBot Google Fetches a page when a person asks an assistant Google-CloudVertexBot
GoogleOther Google AI training crawler GoogleOther
Applebot-Extended Apple AI training crawler robots.txt token only
Bytespider ByteDance AI training crawler Bytespider
CCBot Common Crawl AI training crawler CCBot
Meta-ExternalAgent Meta AI training crawler meta-externalagent
Meta-ExternalFetcher Meta Fetches a page when a person asks an assistant meta-externalfetcher
Amazonbot Amazon AI search crawler Amazonbot
DuckAssistBot DuckDuckGo Fetches a page when a person asks an assistant DuckAssistBot
MistralAI-User Mistral AI Fetches a page when a person asks an assistant MistralAI-User
cohere-ai Cohere Fetches a page when a person asks an assistant cohere-ai
cohere-training-data-crawler Cohere AI training crawler cohere-training-data-crawler
YouBot You.com AI search crawler YouBot
AI2Bot Allen Institute for AI AI training crawler AI2Bot
Diffbot Diffbot AI training crawler Diffbot

This is the list the Wholetally edge collector recognises. We re-check it against the operators’ documentation from time to time.

Check which AI crawlers your robots.txt allows

Count the visitors you’ve been missing.

30-day free trial, no card needed. Create an account with your email and add your first site.

Start 30-day trial