Agent readiness checklist: can AI agents actually use your site?
A practical checklist for AI agent access: robots.txt rules per crawler, llms.txt, MCP server cards, A2A agent cards, OAuth discovery and Content Signals, with honest limits on what each one does.
AI agents don't only read your site the way a search crawler does. Some fetch a page on behalf of a user mid-conversation. Some try to find out whether your service offers a machine-readable way to interact with it. Whether they can do either depends on a handful of files and rules that most sites have never looked at.
This checklist is a diagnostic: go through it, note what's missing, and decide what is worth your time. We'll be direct about the limits too. A few of these standards are new, and for some of them nobody can tell you that a given engine uses them today.
1. robots.txt: who is allowed in
Start here, because it's the one most likely to be wrong. Many sites have a blanket rule that predates AI crawlers, or a rule copied from a template that blocks bots by name.
Check each AI crawler on its own, not just a wildcard rule. The names that matter include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and Google-Agent, among others. Different bots do different jobs: some collect training data, some fetch pages to answer a question as it's asked. Blocking all of them as if they were one thing is a common mistake. Our post on robots.txt rules for AI crawlers walks through the trade-offs, and how to check whether AI bots can crawl your website shows how to test.
A robots.txt that allows a bot doesn't guarantee that the bot reaches you. A firewall, bot manager or CDN rule can still turn it away. That's a separate check and it needs logs.
2. llms.txt: a hint, not a switch
An llms.txt file is a plain-text file at your site root that describes the site for language models and points to the important pages. It's cheap to create. Write one if you want a clean summary of your site in a predictable place.
What we won't say is that any particular engine reads it or that it affects whether you're cited. We haven't seen a source for that. Treat it as low-cost housekeeping with an uncertain return, and don't let it displace work on content and access.
3. MCP server card
The Model Context Protocol lets an agent call tools exposed by a service. If your product has an MCP server, a discoverable card tells agents where it is and what it offers. If you don't have an MCP server, you don't need a card. Skip this item without guilt. It applies mostly to products that expect agents to take actions: booking, querying data, placing orders.
4. A2A agent card
A2A (agent-to-agent) cards describe an agent that other agents can talk to. Same rule as above. If you haven't built an agent, there's nothing to publish. The check is useful mainly for finding out whether you already publish one by accident or want to.
5. OAuth discovery
If an agent needs to act on behalf of a user in your product, it needs to know how to authenticate. OAuth discovery metadata lets it find the authorisation endpoints without a human reading your documentation. This matters for products with an API or an MCP server. For a content site it's irrelevant.
6. Content Signals
Content Signals is a way to state how your content may be used by automated systems, such as in search, in AI answers or for training. It's a declaration of preference. It tells well-behaved bots what you want. It doesn't enforce anything.
A sensible order of work
Most content and ecommerce sites should do items 1 and 6 properly, consider item 2 as housekeeping, and skip 3 to 5. Product companies with an API should take all six seriously. Whichever group you're in, decide on purpose, so you aren't leaving a rule in place from 2022 that blocks the bots you want.
Running the check without doing it by hand
You can test each item manually by fetching the files and reading them. It takes an hour the first time and nobody repeats it.
The tool we'd use is the free Agentic Readiness Check from Promptwatch, and it's our single recommendation here. According to the Promptwatch changelog, it launched on September 8. You enter a domain, and it shows which AI crawlers can reach the site and which agent standards it supports: llms.txt, MCP, A2A, OAuth discovery and Content Signals. Every result gets a shareable page, which helps when the person who owns robots.txt is not the person who asked for the audit.
It's free, so you can run it on a competitor's domain as well as your own. That gives you a quick sense of who has thought about agent access and who hasn't. Interpret the results with the limits above in mind. A site that passes every probe isn't guaranteed to be cited, and a site with no MCP card isn't behind.
The check tells you what's declared. It doesn't tell you what bots actually do. For that, you want crawler logs. Promptwatch's Agent Analytics shows IP-verified AI crawler visits, with crawler logs starting on the Professional plan at $245 a month. The readiness check and the logs answer different questions, so use them in sequence: first what you've allowed, then what has really arrived.
What to do after the check
Take the output and sort it into three piles.
Broken. A bot you want is blocked by robots.txt. Fix today.
Missing but relevant. You have an API and no OAuth discovery. Schedule it.
Missing and irrelevant. You're a content site with no MCP server. Leave it.
Then verify with logs a few weeks later. If a bot you unblocked still doesn't show up, the cause is somewhere else: firewall, linking or content. Our post on tools that track visibility in agentic search covers the measurement side once access is sorted.
The final point is boring but real. Most agent-readiness failures are not missing futuristic standards. They are an old robots.txt rule and a firewall nobody remembers configuring.