The crawl-to-citation path: why fetched pages still go uncited

"But the bots are visiting." We hear this a lot, usually from a technical lead who has checked the logs and found GPTBot and friends all over the site. They're right that access is a precondition. They're wrong if they assume it's enough.

A page has to get through several stages to be cited. Mapping them gives you a way to find where yours stops.

The path, in stages

Think of five stages:

1. Discovery. A crawler learns the URL exists.
2. Fetch. The crawler requests it and receives usable content.
3. Retention. The engine keeps something from the page in a form it can retrieve.
4. Selection. For a given question, the engine picks this page among candidates.
5. Citation. The answer links or names it.

You can observe stages 2 and 5 directly. The others you infer. Even so, the inference is useful, because the pattern of where a page stalls is quite informative.

Promptwatch's view of the path

In Promptwatch, Agent Analytics logs AI crawler requests with IP verification, so spoofed agents are filtered out. It then connects those requests to citations, giving a citation rate per page. It also tracks errors with failure explanations, and lets you stack path filters and export to CSV.

That's the crawl-to-citation path: for each page, did bots come, did the visits succeed, and did any citation follow? We use it as a triage tool.

Pattern 1: no fetches

The page isn't being crawled at all. Causes include blocking in robots.txt or the CDN, no internal links to it, no sitemap entry, or a page that sits behind a login. Fix discovery and access. Our post on AI crawler access lists the checks.

Pattern 2: fetches with errors

Bots arrive but receive 403, 404, 429, 5xx or a timeout. The failure explanations in the logs are worth reading. We've seen cases where a security rule blocked a bot that robots.txt explicitly allowed, and where a redirect chain broke on a mobile subdomain. A fetch that errors can't turn into a citation, so fix these before anything else.

Pattern 3: clean fetches, no citations

This is the interesting one. The bots arrive, the pages return fine, and nothing is cited. The problem is now probably about the content or its fit with the prompts.

Check the following:

• Does the page answer a real question? Look at fan-outs for your target prompts. If engines search for sub-questions your page never addresses, selection will skip it.
• Is the answer easy to lift? A direct statement near the top, a clear heading, a table where comparison is the point.
• Is the page rendered in a form a simple fetcher can read? Content injected late by JavaScript may be missing.
• Is it thin or duplicate? If three of your pages say the same thing, the engine may pick none.
• Is it corroborated? Engines often lean on third-party sources too. A claim that appears only on your own page is weaker than one backed elsewhere.

We don't know the exact selection logic of any engine, and nobody outside the vendors does. What we do is run experiments on single pages and watch.

Pattern 4: cited, but not for the right prompts

The page is selected, but for different questions than you intended. That's not failure. It's information about the page's value. Use the Page Tracker to see the prompts that cite it, then decide whether to adjust the page or write another one for the missing intent.

Pattern 5: cited, but no visits

Citation without clicks is common. The engine used your page as background and the reader never needed to click. Visitor analytics in Promptwatch records AI-referred traffic and conversions, so you can see which cited pages earn visits. If a high-citation page earns none, consider whether the answer is complete enough that no one needs more, and whether a clear next step exists for people who do click.

A worked method

Take your ten most important pages and make a table with columns for fetches, errors, citations and visits. Fill it from the logs and citation views. Within an hour you'll have a map of where each page sits.

Then assign one fix per page, working from the earliest failing stage. Don't rewrite content for a page that has never been fetched. And don't unblock a crawler for a page whose real problem is that it doesn't answer anything.

Why order matters

Teams love to skip to content, because content feels like the lever. But a content rewrite is wasted if bots can't reach the page, and an access fix is wasted if the page has nothing worth citing. The path forces the right order: access first, then fetch quality, then content fit, then distribution.

What the Action Board adds

Findings that stay in a spreadsheet die. Promptwatch's Action Board is a prioritized GEO to-do list. Its Discovery agent investigates the data weekly and suggests up to five evidence-backed actions per run, and you can add your own with assignees and comments. We log each page's next fix there, so a month later we can see whether anything happened.

Measuring the result

After a fix, wait a few weeks and look at the same page. Did fetch errors drop? Did a citation appear? Did visits follow? We don't promise outcomes, and engines don't commit to timelines. But the sequence gives you something concrete to check.

What we'd do first

Connect crawler logs and let Promptwatch assemble the path per page. Then pick the five pages that are fetched but never cited and read them as an outsider would. That one exercise usually surfaces more useful edits than a week of brainstorming.