Can AI crawlers read your location pages?
Location pages are where local AI answers should come from, and they're often the hardest pages on a site for a crawler to read. What to check in robots.txt, CDN rules, rendering, and store locators, and how to verify it in logs.
If an AI engine is going to name your Rotterdam branch in an answer, the most useful page it could read is your Rotterdam branch page. It's also the kind of page that's easiest to build in a way a crawler can't read.
We've written about checking AI crawler access in general: robots.txt, CDN and firewall rules, then logs. Location pages deserve their own pass, because they break in different ways. They're usually generated from a template, often built by a store locator vendor, frequently dependent on JavaScript and a map widget, and sometimes hidden behind a postcode search. Each of those is a place where a crawler can come away with nothing.
Check 1: robots.txt rules on the location paths
Open robots.txt and look at the rules that touch wherever your location pages live: /locations/, /stores/, /branches/, or a subdomain like stores.example.com. A subdomain has its own robots.txt, and it's often managed by the locator vendor, not your team.
The rules that catch location pages by accident are usually broad ones. A disallow on URLs with query parameters can block a locator that builds pages like /stores?city=utrecht. A disallow on a folder used for "internal search" can take the locator results with it.
OpenAI's crawler documentation says each of its robots.txt settings is independent. You can allow OAI-SearchBot, which is the crawler for ChatGPT search, while disallowing GPTBot, which collects training data. Sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers. OpenAI also says a robots.txt change can take around 24 hours to take effect for search. The same logic applies to PerplexityBot, ClaudeBot, and Claude-SearchBot: check each one against the location paths, not just the homepage.
Check 2: CDN and firewall rules
Bot protection is where location pages quietly fail. Some patterns we look for:
- Bot management or "block AI crawlers" settings at the CDN that cover the whole zone, including the locator subdomain.
- Rate limits on the location folder or the locator subdomain. A crawler that fetches a hundred branch pages in a row can trip a limit built for human browsing.
- Country blocking. A business that only operates in one country sometimes blocks traffic from elsewhere, and the crawlers don't necessarily come from your country.
OpenAI recommends that the host or CDN allow requests from its published IP ranges. Google's documentation on AI features lists making sure crawling is allowed "in robots.txt, and by any CDN or hosting infrastructure" as a basic. Ask whoever owns the CDN account; it's rarely marketing.
Check 3: is the content in the HTML?
Fetch a location page the way a simple crawler would, without running JavaScript. A command-line fetch or the browser's view-source will do. Then look for the things an answer needs: the branch name, the address, opening hours, services offered at that branch, and anything that makes it different from the others.
If what you see is a page title, an empty div, and a script tag, a crawler that doesn't render JavaScript sees the same. Many AI crawlers fetch raw HTML without running scripts, so the map, the hours widget, and the reviews carousel may as well not exist. Server-side rendering or static generation for location templates fixes this, and it's usually a template change, not a per-page job.
Check 4: store locators that hide content
Store locators deserve their own check because they combine several problems.
The most common one is a locator that only shows branches after someone types a postcode or allows location access in the browser. Crawlers don't fill in forms or grant permissions, so for them the locator is an empty search box. If there's no other route to the branch pages, those pages are effectively invisible.
Others to look for: branches that exist only as pins in a map with no URL of their own, pagination that loads more results by script, and locators embedded in an iframe from the vendor's domain, which means the content lives on their site and not yours.
The fix has the same shape each time. Every location gets its own URL with its content in the HTML. An index page links to every location in plain links, grouped by region or city if there are many. The sitemap lists them. The locator can stay for human visitors; it just stops being the only door.
Check 5: redirects and interstitials
Two more things to check on the location templates. First, IP-based redirects. If the site sends visitors to a regional version based on their IP, a crawler lands on whatever version its own IP maps to, which may not be the one you want read. Second, overlays. A "choose your nearest store" modal or a location permission prompt that covers the content is a nuisance for people and can be worse for a crawler, depending on how it's built. Check the raw HTML again with the overlay in mind.
Verify it in the logs
All of the above tells you what should happen. Logs tell you what did. Filter your access logs for AI user agents on the location paths and look at three things: whether they're fetching those pages at all, which status codes they get, and whether the fetches cover every branch or just a few.
User agents can be spoofed, so for anything you plan to act on, check requests against the IP ranges the operators publish.
We do this for clients with Agent Analytics in Promptwatch. It records AI crawler visits in real time with IP verification, so spoofed user agents are filtered out, and it tracks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, the Meta crawlers, and others by name. Stackable path filters let us isolate the location folder. Error tracking explains why a fetch failed, which is how a rate limit or a firewall rule on the location folder shows up. The crawl-to-citation path gives each page a citation rate, so we can see whether the Rotterdam page got fetched and then cited, or fetched and ignored.
Crawler logs are on Promptwatch's Professional plan and up and on every agency plan; Essential doesn't include them. Logs come in through Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint, or the WordPress plugin.
If you'd like us to audit your location pages for AI crawler access, write to hello@visiblyrank.com.