Claude-SearchBot: what it is and how to see it

Anthropic’s crawler for search results in Claude.

Operator
Anthropic
How we classify it
AI search crawler
User agent token
Claude-SearchBot
Visible in traffic
Yes, by user agent
Operator documentation
Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?

The one-line description above restates how we classify this crawler. It is not the operator’s wording.

Block or allow it in robots.txt

robots.txt: block
User-agent: Claude-SearchBot
Disallow: /
robots.txt: allow
User-agent: Claude-SearchBot
Allow: /

Rules for one crawler don’t affect the others from the same company.

Why GA4 doesn’t show it

Claude-SearchBot fetches your HTML without running the JavaScript in it, so a browser tag never fires. Crawler requests only show up in server, CDN or edge logs.

How Wholetally counts it

The edge collector (a Cloudflare Worker or Next.js middleware) recognises AI crawlers by user agent and counts their hits per page in the AI report, outside your visitor numbers. It does not check IP ranges, so a bot that pretends to be Claude-SearchBot is counted as Claude-SearchBot.

Others from Anthropic

Count the visitors you’ve been missing.

30-day free trial, no card needed. Create an account with your email and add your first site.

Start 30-day trial