An AI provider is missing from your crawler logs: what to check
Promptwatch's crawler data shows OpenAI at 79.8% of verified AI crawler requests. If another provider is near zero in your logs, here is the order we check things in.
You open your crawler logs for the first time. OpenAI is there, busy. Anthropic shows a trickle. Perplexity is a handful of requests, and Mistral isn't there at all. The natural question is whether you are doing something wrong, or whether that's just how it looks.
Promptwatch's AI crawler traffic report gives a reference point. In the week of August 31 to September 6, 2026, OpenAI accounted for 79.8% of verified AI crawler requests, down from 94.8% in the week of June 8 to 14. The providers counted are OpenAI, Anthropic, Google, Perplexity and Mistral. The page now updates weekly, so the figures quoted here are the ones from that week, not today's.
This note is the order of checks we use when one of those providers is missing from a site's logs.
Read the report carefully before comparing
Three details change how you should use the numbers.
Shares, not volumes. Each week's shares total 100%. A provider's slice can grow without it sending more requests, if others send fewer. OpenAI's share falling from 94.8% to 79.8% means other providers' slices grew relative to it. It doesn't tell you how many requests anyone sent.
Provider totals hide the type of crawler. A provider's share lumps together training crawlers and answer-time fetchers. The report's chart can group by crawler, and that split is where the meaning is. Heavy OpenAI traffic could be training, answer-time fetches or both.
Only verified requests count. The report counts requests whose IP address matches the operator's published ranges. Visits that merely claim a user agent such as GPTBot don't count. Weeks run Monday to Sunday, UTC, and the data comes from raw server and CDN logs connected to Promptwatch crawler analytics.
The last point matters for your own comparison. If your log tool counts by user agent only, it will include spoofed visits, and your numbers won't be comparable.
The test the report suggests
The report's advice is to compare your own logs with this mix. A provider that is large in the report but close to zero in your logs suggests you may be blocked or unreachable for it.
"Suggests" is the right word. A small site can receive less crawler traffic than the aggregate implies, for reasons unrelated to blocking. Your site may be small, new, rarely linked, or in a niche a provider doesn't crawl often. We don't have data on why any individual site is crawled less, and the report doesn't claim one. So treat a gap as a reason to run the checks below, not as a diagnosis.
The checks, in order
1. Read robots.txt as each bot would. Open your robots.txt and look for rules naming each provider's user agents. The real names are GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI; ClaudeBot, Claude-User and Claude-SearchBot for Anthropic; PerplexityBot and Perplexity-User for Perplexity; MistralAI-User for Mistral; and Google-Agent and GoogleOther among Google's. Our list of AI crawler bot names says what each does. A wildcard Disallow or a blanket rule left over from a scraper scare can catch all of them. Our piece on robots.txt rules for AI crawlers covers changing it carefully.
2. Check your CDN and firewall. This is the usual culprit when robots.txt is clean. Bot-management features, rate limits and country rules can return a 403 or a challenge page to a crawler without anyone deciding to block it. Find the rule set, look at bot categories, and look at what is logged as blocked.
3. Test from the outside. Request a page with a different provider's user agent string and watch the status code. It won't replicate IP-based rules, because your test comes from the wrong address, but it will expose user agent rules. Our walkthrough on whether AI crawlers can reach your site shows the sequence.
4. Look at the logs you are comparing. Are the logs complete? Cached responses served at the edge may never reach your origin, so origin logs can miss visits that a CDN log would show. Check the retention period and any sampling too. A provider that visits rarely will vanish from a sample.
5. Check the status codes. A provider present in the logs but receiving 403, 404 or 5xx responses is a different problem from one that never arrives. The fix list differs, and you only see it if the logs carry status codes.
6. Look at the pages. Is the content reachable by a plain request, without a login, a cookie wall or a script-only render? A crawler that arrives and finds an empty page won't come back often.
7. Wait, and re-measure. One week of logs is thin evidence for a provider that visits infrequently. Look at four weeks before you conclude anything.
What we wouldn't conclude
We wouldn't say a missing provider means you are invisible in that provider's answers. Crawl access and citation are separate steps, and the logs only cover the first one. Nor would we say that unblocking a crawler gets you cited. We'd only say that a page a provider cannot fetch is one it cannot cite from your site.
Measuring it without building a log pipeline
You can do all of the above by hand, but verifying IPs against each provider's published ranges and keeping them current is the part people drop. We run this on Promptwatch, using Agent Analytics: real-time, IP-verified crawler logs that filter out spoofed user agents. You can filter by crawler, which gives the training-versus-answer-time split the report talks about, and error tracking explains failures. The crawl-to-citation path shows, per page, whether fetched pages go on to be cited, which we cover in a separate note on pages fetched but never cited.
Setup runs through Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, custom HTTP, or the WordPress plugin. If you have no server access, our guide on seeing AI crawler visits without root access goes through the options.
On plans: crawler logs start on Professional at $245 a month, and Essential at $95 a month has none. Business includes crawler logs at higher volume, and every agency plan includes them, starting at Kick-off at $199 a month. If you are on Essential, the manual checks above are your route.
The routine
Put the week's provider mix from the report next to your own. If a provider is missing, run checks one to three, which cover the usual places a block hides. If it's present but erroring, go to check five. If everything is open and the provider is still quiet, note it and re-measure in a month. That's often all the data supports.