AI crawler bot names, decoded: what each one is and what to do with it

Open a week of server logs and you'll find a zoo of user agents. Some are familiar. Many are new. When a client asks which AI bots are hitting their site, we start by putting names on the ones that matter and leaving the rest.

This is the working list we use, and the rules for acting on it.

The names Promptwatch tracks

Promptwatch's Agent Analytics logs a defined set of AI crawlers and agents:

• GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI
• ClaudeBot, Claude-User and Claude-SearchBot from Anthropic
• PerplexityBot and Perplexity-User
• Google-Agent and GoogleOther
• Meta-WebIndexer, Meta-ExternalAgent and Meta-ExternalFetcher
• MistralAI-User

There is no crawler called "ChatGPTBot". If you see that string, somebody is guessing or faking it. The platform's AI Bots directory lists more than 440 crawlers, which tells you how long the tail runs.

Three kinds of bot activity

We don't have verified purpose statements for every bot above, so check each vendor's current documentation before writing rules. What we can describe is a pattern that shows up in naming:

1. Collectors that fetch content in bulk, sometimes for training or indexing.
2. Search or index bots that build the pool of pages an assistant can retrieve from.
3. User-triggered fetchers that load a page because someone asked a question that needed it. Names containing "User" often fall in this group, but confirm with the vendor.

That split is useful because the decision differs. You might be comfortable letting a user-triggered fetcher read your public pages while having a different view on bulk collection.

User agents are easy to fake

Anyone can send a request claiming to be a famous bot. If you build rules purely on user agent strings, you can end up counting scrapers as AI crawlers, and you might allow or block the wrong traffic.

That's why Promptwatch's crawler logs are IP-verified and filter out spoofed user agents. If you do your own log analysis, check requests against the vendor's published IP ranges where they exist. A page that "was fetched by GPTBot 400 times" from an unverified address might not have been fetched by GPTBot at all.

Reading logs for the pages you care about

Raw bot counts are vanity. Ask page-level questions.

• Does an AI search bot request the pages I want cited?
• Do the requests succeed? Look for 403, 404, 429, 5xx and timeouts.
• How often? A page crawled once a month behaves differently from one crawled daily.
• Does a fetch ever lead to a citation?

Promptwatch's crawl-to-citation path puts a citation rate on each page, and error tracking explains failures. You can stack path filters and export to CSV. Crawler logs start on Professional for brands (Essential has none) and are included on every agency plan.

Deciding who gets in

A sensible default for most brands that want AI visibility is to allow the search and user-triggered bots on public pages that you want cited, and to make a conscious call on bulk collectors. Write each decision down.

Some points we've learned the hard way:

• A blanket "Disallow: /" for all unknown agents can lock out assistants you wanted.
• A WAF or CDN rule can block a bot even when robots.txt allows it. Both layers need checking.
• Rate limiting can return 429 errors to legitimate bots and look like a content problem.
• Pages behind login, heavy JavaScript rendering or cookie walls may return nothing useful to a simple fetcher.

Our full walk-through is in can AI crawlers reach your site. If you need a refresher on what robots.txt actually enforces, remember it's a request, not a lock.

Connecting your logs

You need a log source before any of this works. Promptwatch supports crawler logs through Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, custom HTTP, or the WordPress plugin. Pick what sits closest to your origin. A CDN-level feed sees bots that never reach your server.

A short audit

Here is the check we run in an hour:

1. Pull seven days of logs or open Agent Analytics.
2. List which AI bots visited, verified by IP.
3. For the ten most important pages, note bot visits and error codes.
4. Compare with robots.txt and CDN rules.
5. Fix errors for bots you intend to allow.
6. Record the policy decision for each bot.

Then re-check in two weeks. Fixes often reveal a second layer of blocking.

Mistakes we see

The most common: a developer adds a bot-blocking rule during a scraping incident and nobody remembers. Six months later the brand wonders why it vanished from answers. Others include blocking every bot whose name contains "AI" and leaving staging rules in production.

The cure is a short, dated policy document and a log review on a schedule, not a one-off panic.

Our recommendation

For real visibility you need the names, the verification and the page-level view together. That's what Promptwatch provides through Agent Analytics: real names, IP-verified, with the crawl-to-citation path. We'd start there, write down the policy, and revisit after the next quarter's logs. It's a modest amount of work, and most of the benefit comes from finally knowing what's actually visiting.