How to track GPTBot and other AI crawlers on your site

Short answer

To track GPTBot and other AI crawlers, count requests on the server side: in your web server or CDN logs, or with an edge collector in front of your site. Group them by user agent and by purpose (training, search, user request). A JavaScript analytics tag such as GA4 will not show them, because crawlers fetch your HTML without running the scripts in it. Verify any bot that matters against the IP ranges its vendor publishes, since a user agent string can be faked.

Are AI crawlers the same as AI referrals?

No, and mixing them up is the most common mistake in AI traffic reports. A referral is a person who clicked a link in an answer and landed on your site; we cover that side in how to see ChatGPT traffic in your analytics. A crawler request is a program fetching a page: to train a model, to build a search index, or because someone asked an assistant a question that needed your page.

The two answer different questions. Referrals tell you who sends visitors. Crawler requests tell you which of your pages AI systems read, how often, and for what. A page can be read by GPTBot a hundred times without sending a single visitor, and the reverse happens too. Put both on one chart and neither number means anything.

Why do GA4 and other browser tags never see GPTBot?

A browser tag works like this: your page loads, the browser executes a script, and the script reports a pageview. Crawlers stop before that last step. They request the HTML, read it, and leave. No script runs, so no hit is ever sent. A tag-based tool has no event to count, which is why GPTBot is absent from GA4 reports instead of merely being filtered out.

Many tools also drop known bots on purpose, so “my analytics shows no bots” is not evidence that no bots visited. The only place the request exists is on the path between the crawler and your origin: your web server’s access log, your CDN’s logs, or a collector at the edge.

Which AI crawlers should you look for?

Each vendor publishes a short list of user agents and states what each is for. Based on their documentation (links in the sources below, checked 7 November 2026):

• OpenAI. GPTBot crawls content that may be used to train its models. OAI-SearchBot surfaces sites in ChatGPT search. ChatGPT-User visits a page when a user action in ChatGPT triggers it, and OpenAI notes that robots.txt rules may not apply to it because a person initiated the visit. OpenAI says each robots.txt setting is independent.
• Anthropic. ClaudeBot collects content that could contribute to training. Claude-SearchBot crawls to improve search results. Claude-User may visit sites when someone asks Claude a question.
• Perplexity. PerplexityBot is described as designed to surface and link websites in Perplexity search results, and not as a crawler for foundation-model training. Perplexity-User supports user actions and, because a person triggered it, generally ignores robots.txt rules.
• Google. Google-Extended is a robots.txt token that controls whether content Google crawls may be used for Gemini training and grounding. Google documents that it has no separate HTTP user agent string, so it never appears in your logs. Google’s crawling still shows up as Googlebot and other existing Google user agents.

Group these by purpose rather than by company. “Training” volume tells you what is being collected; “search” volume tells you what an assistant can cite; “user request” tells you which pages people are asking assistants about. Three buckets are far more useful than one long list of bot names.

How do you track them from your own logs?

If you can read access logs, you can start today with no new tool:

1. Export a few weeks of logs from your web server or CDN. You need the timestamp, the path, the status code and the user agent.
2. Filter on the user agent substrings above (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) and count requests per day.
3. Count requests per path to see which pages are read. Ignore asset files; look at HTML responses.
4. Check status codes. A crawler that receives mostly 403 or 404 responses is not reading what you think it is reading.
5. Compare against Googlebot and Bingbot in the same window, so you can see AI crawlers relative to classic search.

The weak spots are retention and effort. Many hosts keep logs for days, not months, and a one-off script is not a report you will open every week. Cloudflare takes the dashboard route on its side: its AI Crawl Control shows which AI services are accessing your site and lets you set allow or block policies per crawler, if your site runs on Cloudflare.

Can you trust a user agent that says “GPTBot”?

Not on its own. A user agent is a header the client chooses, so any script can claim to be GPTBot. That does not make user-agent counting useless; for trend analysis it is usually good enough. It does mean you should not make security or blocking decisions on it.

The vendors publish ways to check. OpenAI provides IP range files for GPTBot, OAI-SearchBot and ChatGPT-User. Perplexity publishes one for each of PerplexityBot and Perplexity-User and calls them the source of truth. Anthropic links to a list of its crawler IPs and states that a crawler whose source IP is on that list comes from Anthropic; it also warns that blocking by IP can stop its bots from reading your robots.txt, so use robots.txt to opt out. Google lists its common crawlers’ IP ranges and reverse DNS patterns, and itself warns that the user agent string can be spoofed.

In practice: count by user agent for reporting, and verify by source IP wherever a wrong answer has a cost, such as an allow rule in your firewall or a dispute about who is hammering your server.

Why does an edge collector work better than logs for this?

An edge collector sits in front of your site, sees each HTML request the way a log would, and sends it to your analytics. That keeps the benefit of server-side counting and removes the log wrangling.

Wholetally’s edge collection is a one-file Cloudflare Worker, or a middleware for Vercel and Next.js. For each HTML page it serves, it sends one small beacon after the response has gone out, so visitors do not wait for it. The beacon carries the path, the user agent, the country header your CDN already adds and a request ID; it does not carry the IP address. The what we collect page lists every field. Crawlers are recognised by user agent and go to the AI report instead of your visitor numbers, so a GPTBot spike never inflates “visitors”.

The AI report shows requests per bot, split into training, AI search and user request, and which pages each bot read. It sits next to the human side: visitors who arrived from ChatGPT, Perplexity, Claude, Gemini and Copilot, as a separate channel. You can check the recognised list on the page itself; a crawler that is not on it shows up as an unknown bot, not as a person.

What will this not tell you?

• Spoofed and unlisted bots. Wholetally matches by user agent and does not check IP ranges, so a bot pretending to be GPTBot is counted as GPTBot. Bots that are not on the recognised list appear as unknown bots.
• Cached pages. Pages served by a CDN cache without passing through the Worker or middleware are not seen by the edge. Put the collector on the route that serves the HTML.
• Google-Extended and Applebot-Extended. These are robots.txt tokens, not separate crawlers in traffic, so there is nothing to count.
• What happens after the fetch. A request in your log shows that a bot read a page. It does not show whether the page was used for training, quoted in an answer, or discarded.
• Robots.txt behaviour. You can see whether a bot you disallowed still appears, but a request alone does not tell you why.

What can you do with the numbers?

Three decisions get easier once crawler data is separate from visitor data. First, robots.txt: see whether a bot you blocked still shows up, and whether the ones you allowed actually come. Second, content: if search crawlers such as OAI-SearchBot or Claude-SearchBot never request your pricing page, an assistant has little chance of citing it. Third, reporting: tell a client or a boss how much of “traffic” is bots, and keep that out of the sessions chart.

What to do next

If you have log access, run the filter above this week and count requests for the eight user agents. If you would rather have it as a report that updates itself, install the edge collector and open the AI report. Referrals work with the script alone; the crawler side needs the edge collector.

New accounts get a 30-day trial, no card needed. Start on Pricing, or go straight to the app sign-up linked from that page.

Wholetally is cookieless web analytics with an AI report, edge collection and revenue per source. Built in the Netherlands. This article is general information, not legal advice.