Writing robots.txt rules for AI crawlers: a careful approach

Robots.txt is a short file with outsized consequences. A single line can remove a whole site from a crawler's reach, and the person who wrote it is often gone by the time anyone notices. With AI crawlers, the stakes are murkier than they were with classic search, because the bots have different jobs and teams have different appetites for being used.

Here is the process we follow before touching the file.

Decide the policy before the syntax

Rules follow from a decision, not the other way round. Write down three things.

First, do you want to be cited and recommended by AI engines? For most businesses selling something, yes.

Second, are there content areas you'd rather keep out of AI systems, such as paid material, member areas or draft content?

Third, who owns the file? Name a person. If nobody owns it, it will drift.

We've seen teams skip this and paste a template from a blog post. That's how you end up blocking a bot you wanted.

Know what robots.txt can and can't do

It's a request. Well-behaved crawlers honor it. It doesn't enforce anything, and it doesn't stop a determined scraper. It also doesn't remove pages that have already been collected.

Separately, your CDN, firewall or bot-management layer may block requests regardless of what robots.txt says. So a page can be allowed in the file and still unreachable. We've covered this layered problem in can AI crawlers reach your site.

Use real bot names

Each rule targets a user agent name. Use the exact names, taken from vendor documentation. Promptwatch tracks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher and MistralAI-User. Don't write rules for "ChatGPTBot", which doesn't exist.

Check each vendor's current documentation for what each bot does and which user agent token to use in robots.txt, since we haven't verified those details here and they can change. Be aware that some vendors use one token for several purposes, and others split them.

Rules that backfire

A few patterns we flag whenever we see them:

• Blanket disallow for unknown agents. A wildcard rule that blocks everything except a short allowlist can shut out assistants that launch after you wrote it.
• Disallowing the whole site to stop one folder's scraping. Narrow the path.
• Leaving a staging rule in production. Check after every deploy that touches the file.
• Rules that contradict each other. Different crawlers resolve conflicts differently, so avoid ambiguity.
• Blocking CSS and JS directories. It can stop renderers from understanding the page.

Keep the file short and commented. Each comment should say who added the rule and when.

A cautious structure

We structure the file in sections. A default section for general crawlers. A section for each AI bot we've made a decision about, with a comment on why. A sitemap line at the bottom.

For bots you want to allow, an explicit allow section beats silence, because it documents the intent. For bots you want to block, add the rule and record the reason and review date. Treat a block as a decision with an expiry, not a permanent fixture.

Test, then watch the logs

After any change, test the file with a robots.txt tester or by fetching it as the crawler would. Then verify in logs that the behavior matches what you intended.

This is where most teams stop too early. They edit the file and assume. Logs tell you whether the bot you allowed started arriving and whether it succeeds.

Promptwatch's Agent Analytics gives real-time, IP-verified crawler logs, so spoofed agents don't confuse the picture. You can filter by path, see errors with failure explanations and export to CSV. Crawler logs are on Professional and above for brands, and on every agency plan.

What about llms.txt

People often ask whether llms.txt replaces robots.txt. It doesn't. It's a different file with a different purpose and, as far as we know, no guarantee that any engine reads it. We explain this in llms.txt: what it does and what it doesn't do. Keep robots.txt accurate regardless. Promptwatch offers free llms.txt and robots.txt generators if you want a starting point, but review whatever they produce against your own policy.

Version control and review

Put robots.txt in version control. Require a review for changes. Add a calendar reminder to read the live file every quarter. Compare it to the repository version, since CDN or CMS plugins sometimes inject their own rules.

For sites on WordPress, check SEO and security plugins too. Some of them edit robots.txt dynamically, and the file you see in the repo may not be the file bots receive.

When citations drop after a change

If your AI citations fall after a robots.txt edit, don't panic and don't assume the edit caused it. Check these in order:

1. Does the live file match what you intended?
2. Do crawler logs show a drop in visits from the affected bot?
3. Are error rates up?
4. Did anything else change, such as CDN rules or page URLs?

Then compare with your citation data. Citation analytics shows which pages lost citations, and the crawl-to-citation path shows whether a fetch ever converted. If the bot stopped visiting and citations followed, you have a likely cause. If the bot kept visiting and citations still fell, look at content.

The commitment we'd make

Keep one owner, one dated policy, and a quarterly log review. Use Promptwatch to see what actually arrives, because the file only tells you what you asked for and the logs tell you what happened.

That small loop prevents the most expensive kind of mistake, which is the quiet one nobody notices for months.