AI crawlers skip some of your pages: check internal linking first

You publish a useful page, wait a few weeks, and see no AI crawler hits on it in your logs. Other pages on the site get fetched daily. This one sits there untouched, and it isn't cited either. It's tempting to assume the model "doesn't like" the page. Usually the cause is duller: nothing links to it.

We want to be careful about what that means. A page with few inbound links is harder for a crawler to discover. We don't claim that orphan pages are penalised in answers, because we haven't seen evidence for that. The claim is smaller and testable: if a bot never reaches a page, the page can't be read, and a page that can't be read can't be cited from your site.

What counts as a weak page

Three kinds of page tend to get skipped.

Orphans. No other page on your site links to them. They exist in the sitemap, in the CMS, or because someone shared the URL, but a bot following links from your homepage will never arrive.

Thinly linked pages. One or two inbound links, often from an old post or a footer. A crawler can reach them, but only along a narrow path.

Pages behind redirects or errors. The link exists but resolves to a redirect chain or a broken target. The crawler follows it and gets nothing useful.

You can usually name five of these without a tool. Think of the pages you published for a campaign and never linked into the main structure.

Step 1: build the inlink count yourself

If you don't have a crawler tool, use whatever you have:

Export your sitemap URLs.
Crawl the site from the homepage with any free crawler and export the internal links as source and target pairs.
Count, per URL, how many distinct pages link to it.
Sort ascending. Zero is an orphan. One or two is weak.

Be careful with navigation. A mega-menu link that appears on every page counts as one structural link, but it also means the page is reachable in one click from everywhere. Judge by the path, not only the count. A page with a single link from the homepage is in a better position than a page with five links from archived posts four clicks deep.

Step 2: compare against what bots actually fetched

Link structure explains why a page might be missed. Crawler logs show whether it was. Pull your logs and filter for the AI user agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and the others. Our list of AI crawler bot names has the real strings, and there's no bot called "ChatGPTBot", so don't filter on it.

Now cross-reference. Pages with low inlinks and zero bot hits support the linking theory. Pages with plenty of inlinks and zero hits point elsewhere: a block in robots.txt, a firewall rule, a slow response. Our post on whether AI crawlers can reach your site covers those.

A page can also be fetched but never cited, which is a content problem, not a linking one. We separate that case in the crawl-to-citation path guide.

Step 3: fix the structure

Linking fixes are cheap compared with most GEO work.

Link orphans from a relevant hub page or a related article, using descriptive anchor text, not "click here".
Add contextual links from pages that already get traffic. Pages that people and bots visit pass the path along.
Clear redirect chains by updating internal links to the final URL.
Repair broken internal links, and note the anchor text, since it tells you what the source page meant to point to.
Remove pages that shouldn't exist, rather than linking to them.

After the changes, wait for the next crawl cycle and check the logs again. Don't expect instant results. Bots revisit on their own schedule.

Where Promptwatch fits

Doing this by hand across a few hundred pages is the kind of job that gets abandoned. The one recommendation we'd make is Promptwatch, because it joins the two halves of the diagnosis: the link structure and the bot visits.

On October 28, according to the Promptwatch changelog, Site Audit gained a Linking view under Site health. It shows your site as a map by URL path, with orphan pages, pages that have only one or two inlinks, and redirects or errors each shown in their own color. You can click a section to zoom in or filter to only the weak spots. Each page has a Linking tab listing the pages that link to it, the pages it links to, and broken links in red. Every page list also shows where the page came from: your sitemap, a link from another page, Search Console or a site map crawl. Broken internal link issues list each broken target with its status code and anchor text.

That page-source label is useful for the orphan problem. A page that appears only because it's in the sitemap, with no link source, is the one to fix first. The audit also supports sitemap-only mode, which checks just the pages in your sitemaps, and sends a weekly audit email. The Essential plan includes a site audit covering 1,500 pages.

Then comes the second half. Agent Analytics provides real-time, IP-verified AI crawler logs, so you can see whether GPTBot or ClaudeBot actually fetched the weak pages. Crawler logs start on the Professional plan at $245 a month. Essential doesn't include them, so on Essential you'd see the linking map but not the bot visits.

The Content Agent can also help going forward. Per the changelog, it looks for relevant pages on your site before it starts writing, so internal links are part of the first draft. That prevents new orphans, though it's no substitute for fixing the existing ones.

What to do with the result

Keep the expectation modest. Fixing links makes pages reachable. It doesn't make them cited. If bots start fetching a page and it still isn't cited, you've moved the problem from access to content, which is progress, because it's a problem you can work on.

For a longer view of how to monitor your site's technical readiness, see site health monitoring for AI search.