Guides · robots.txt for AI bots

How to control AI bots with robots.txt

robots.txt is the most widely supported way to tell automated agents which parts of a site they may request. AI companies that publish crawler documentation generally state that they honour robots.txt for their named agents. That makes it the first place to look when you want to allow, limit or block AI access.

This guide covers practical patterns. It is not legal advice and does not replace reading each operator’s current documentation.

How robots.txt matching works (short version)

A robots.txt file contains groups of rules. Each group starts with one or more User-agent: lines and then Allow: and Disallow: directives. The most specific matching rule for a given path wins; when specificity is equal, Allow often takes precedence depending on the parser. Order of groups and exact string matching matter.

For AI crawlers the important practical points are:

• Use the exact User-agent token the operator documents (for example GPTBot, not a guess).
• A rule for User-agent: * applies to agents that do not have a more specific group.
• Some AI controls (notably Google-Extended and Applebot-Extended) are tokens that affect how content may be used even when the traditional search crawler is allowed.

Block a specific AI crawler site-wide

To disallow a named crawler from the entire site:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Place these groups above or alongside your existing rules. Test with a tool that parses robots.txt according to the standard; Movio reports the exact rule that matched for each crawler it checks.

Allow a crawler only on selected paths

If you want an AI product to see public documentation or a specific section but not the rest of the site:

User-agent: GPTBot
Allow: /docs/
Allow: /blog/
Disallow: /

Specificity rules mean that longer matching prefixes usually win. Confirm behaviour with a live check after you deploy the change.

Google-Extended and similar tokens

Google-Extended is not a traditional crawler User-Agent that you will see making requests under that name in the same way as GPTBot. It is a robots.txt control that influences whether Google may use content for certain generative AI features. A typical block looks like:

User-agent: Google-Extended
Disallow: /

Applebot-Extended works on a similar principle. Blocking the Extended token does not by itself block the main search crawler (Googlebot / Applebot). Read Google’s and Apple’s current documentation before relying on these rules for policy compliance.

Common mistakes

• Assuming robots.txt is enforced by the server. It is not. Non-compliant bots can ignore it. Enforcement at the network layer is a separate step (see the firewall guide).
• Typo in the User-agent name. GPT-Bot or gptbot may not match the official token. Use the spelling the operator publishes.
• Only updating robots.txt and never verifying. CDN caches, multiple robots.txt locations, and conflicting plugin settings are common. After a change, run a check from an external tool.
• Blocking * and expecting AI crawlers to be caught by it. Many AI operators register specific User-agent names; a specific group for that name overrides the wildcard group.

Verifying your rules

After editing robots.txt:

1. Confirm the file is served at https://your-domain/robots.txt with a 200 status and the expected content.
2. Use Movio’s AI Crawler Firewall Checker to see, for each tested crawler, which rule matched and whether a live request was allowed or blocked at the server.
3. If the robots.txt result is “allowed” but the server result is a 403 or challenge, the block is happening at the firewall/CDN layer, not in robots.txt.

Keeping rules maintainable

As new AI crawlers appear, a long list of individual Disallow groups becomes hard to audit. Some teams maintain a short comment block at the top of robots.txt listing the policy intent (“block training crawlers, allow retrieval crawlers that cite sources”) and then keep the technical rules in sync with that intent. Review the list a few times a year against operator documentation.

robots.txt is necessary but not always sufficient. For defence in depth, combine it with network controls described in the next guide.

Check your robots.txt live