Can AI crawlers reach your site? How to check robots.txt, CDN rules, and logs
If GPTBot, OAI-SearchBot, ClaudeBot, or PerplexityBot can't fetch your pages, they can't cite them. How to check access in three layers, and what to fix.
Most AI SEO advice is about content. That makes sense, but it skips a step. Before any engine can quote your page, a crawler has to fetch it, and it's easy to block the bots you most want reading your site without anyone noticing.
We check access in three layers.
Layer 1: robots.txt
Start with the file everyone knows. Open yoursite.com/robots.txt and look for rules that apply to AI user agents. These are the ones we check first:
| User agent | Operator | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects content for model training |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results |
| ChatGPT-User | OpenAI | Fetches pages on demand during a conversation |
| ClaudeBot | Anthropic | Crawls for Anthropic's models |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers |
| Google-Extended | A robots.txt token for Gemini apps and Vertex AI use, not a separate crawler |
Two mistakes are common. The first is a blanket User-agent: * disallow on folders that hold important content, like /docs/ or /pricing/. The second is blocking every AI bot in one go when the intent was only to opt out of training. Blocking GPTBot is a decision you can defend. Blocking OAI-SearchBot as a side effect means opting out of ChatGPT search citations.
Google-Extended is its own case. Google says it doesn't affect inclusion in Search, so it won't change whether you appear in AI Overviews. Those follow normal Googlebot crawling.
Layer 2: CDN and firewall rules
This layer is the one teams forget. CDNs and web application firewalls can challenge or block a bot before robots.txt even comes into play. Cloudflare, for example, offers a setting that blocks AI crawlers across a whole zone, and bot protection features can serve challenges that a crawler can't pass. Other CDNs have similar controls.
Things to check:
- Any "block AI bots" or bot management setting at the CDN level
- Custom firewall rules that match on user agent strings
- Rate limits that could throttle crawlers on large sites
Ask whoever owns the CDN account. Marketing teams often don't have access and may not know these settings exist.
Layer 3: server logs
Rules tell you what should happen. Logs tell you what did. Pull your access logs, or your CDN's logs, and filter for the AI user agents above. You're looking for four things.
Whether they visit at all. A month with no hits from OAI-SearchBot is worth chasing down.
Which pages they fetch. Bots may spend their time on old blog posts and never touch your product or pricing pages.
Status codes. 403s, 429s, 5xx errors, and long redirect chains all waste a crawl.
What they can read. Many AI crawlers fetch raw HTML without running JavaScript. If your content arrives client-side, the crawler may see a near-empty shell.
User agents can be spoofed, so for anything important, check the request against the IP ranges the operators publish. OpenAI, for example, publishes IP lists for its crawlers.
Fixing what you find
The fixes are usually simple once you know the cause: an allow rule for OAI-SearchBot and PerplexityBot, a CDN exception, a redirect cleanup, or server-side rendering for the pages that matter. The hard part is noticing in the first place, because nothing visibly breaks. The site works fine for people and for Googlebot.
Keeping an eye on it
A one-off log export is a snapshot. Bot behavior changes, someone flips a CDN setting, a deploy breaks rendering on a template. We monitor this continuously for clients with Agent Analytics in Promptwatch, which reads AI crawler logs in real time through integrations with Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, and others. It shows which bots hit which pages, flags errors, and connects crawls to citations, so you can see when a page that got fetched later appeared as a source in an answer.
When access breaks, it shows up there first, well before it shows up as a dip in your visibility numbers.