Meta AI and Meta-WebIndexer: what to check

Meta is easy to ignore in an AI search plan. It is not the engine most buyers name when you ask where they look. But two things changed recently, and both are worth ten minutes of attention: Meta AI became trackable as a model, and a Meta crawler became a large share of AI bot traffic.

Meta AI as a tracked model

Since October 28, Promptwatch can track Meta AI as a model. It sits with the real-UI platforms in the tracked list, next to ChatGPT, Gemini, Perplexity, and Microsoft Copilot.

That does not mean you must add it. It means you can find out. Our approach is to add Meta AI to a single monitor first, with a small set of prompts from the existing panel, and look at the answers before deciding. Some clients find their brand is named there in ways they did not expect. Others find nothing, which is also a result. Either way you have an observation instead of an assumption.

A caution on reading it. Meta AI is one more source of variation, and a score for a single engine is not comparable to one for another without care. Report it separately.

The Meta-WebIndexer numbers

Promptwatch's data report on Meta-WebIndexer, at promptwatch.com/data/meta-web-indexer, found that it grew from about 2% to about 38% of tracked AI crawler requests. That is the one figure we rely on here, and it is worth stating exactly what it says: a share of tracked AI crawler requests, in Promptwatch's data. It does not say how many of your visits it accounts for, and it does not say what Meta does with the pages.

What it does say is that if you have not looked at Meta's bots in your logs, there may be more traffic from them than you think. On November 4, Promptwatch's GEO 360 started counting search indexers, including Meta-WebIndexer, as AI crawls, which will move your totals. Read our note on indexers versus user fetches before comparing months.

Real bot names to look for

Use the actual names. They matter for robots.txt and for reading logs.

Meta-WebIndexer. A search indexer, and the one behind the growth figure.
Meta-ExternalAgent. Appears in the list of AI crawlers Promptwatch tracks.
Meta-ExternalFetcher. Also on the tracked list.

We deliberately say little about what the last two do. We have a name and a place on a tracked list, and we will not write a function for a bot we have not read documentation for. Check Meta's own crawler documentation before you write a rule that depends on the purpose.

What to check on your site

Read your robots.txt by bot name. Look for each Meta user agent. A wildcard rule or a copied template can block them without anyone meaning to.
Decide on purpose. Allowing or blocking is a business choice. We have clients who allow all of them and clients who do not. The wrong state is the accidental one.
Test with the free tool. Promptwatch's Agentic Readiness Check tests robots.txt against nine AI crawlers and also looks at llms.txt, MCP server cards, A2A agent cards, OAuth discovery, and Content Signals. We do not assume its nine include every Meta bot, so confirm that against the result and check any missing bot by hand.
Check the CDN and firewall. A permissive robots.txt does not stop a bot manager from turning a crawler away. Logs show it.
Look in the crawler logs. Agent Analytics records AI crawler visits verified against operator IP ranges, so a fake Meta user agent should be filtered out. Look at which pages Meta bots actually reach.

What to do with a clean result

If robots.txt allows the Meta bots, the logs show visits, and you have decided that is what you want, the work is done. Put a note in your technical documentation and review it each quarter. Bot lists change, and a rule you wrote last year may not cover a new user agent.

If you decide to block, make sure the rule matches the name exactly, and confirm in the logs that the visits stop.

What we do not claim

We do not say that allowing a Meta bot leads to mentions in Meta AI, or that blocking it removes you. We have no evidence of that link and the data report does not make it. We say that visibility into the bots and the model costs little, and that surprises in either are worth finding early.

For a free first look at your own access, try the Agentic Readiness Check, or start a 7-day free trial of Promptwatch to see Meta AI and the bots in one place. If you want help reading the results, write to hello@visiblyrank.com.