Blocking AI crawlers at the firewall and CDN layer
robots.txt is a request, not a technical barrier. Compliant crawlers follow it; non-compliant ones can ignore it. Even for compliant agents, site owners sometimes want a hard block: if the User-Agent matches a known AI crawler, return 403 or drop the connection before the application runs.
That hard block happens at the network or edge layer — CDN, WAF, reverse proxy, hosting security plugin, or server configuration. This guide explains how those layers interact with AI crawlers and how to verify the result.
Why the server layer matters
Movio frequently sees the following pattern:
- robots.txt explicitly
Allows a crawler (or has no specific rule). - A live request with that crawler’s User-Agent receives HTTP 403, a challenge page, or a timeout.
The opposite also occurs: robots.txt Disallows the crawler, but the server still returns 200 because nothing at the edge is enforcing the policy. In both cases, looking at only one layer leads to the wrong conclusion.
Server-side blocks also catch agents that do not honour robots.txt. That is the main reason some operators combine both controls.
Where blocks are typically configured
CDN and bot-management products
Cloudflare, Fastly, Akamai, AWS WAF and similar products can match on User-Agent strings, ASNs, or behavioural signals. Many have managed rules or custom rules that treat “AI scrapers” as a category. Defaults vary: some plans are permissive until you enable a rule; others ship with aggressive bot scores that can affect legitimate AI crawlers.
If you use Cloudflare, for example, Bot Fight Mode, Super Bot Fight Mode, and custom WAF rules can all return challenges or blocks to requests that present AI User-Agents. The exact behaviour depends on your plan and configuration. Movio can detect some Cloudflare challenge indicators in responses; it cannot disable or bypass them.
Hosting and application security plugins
WordPress security plugins, cPanel/ModSecurity rules, and host-level “bot protection” features often maintain lists of undesirable User-Agents. AI crawler names are increasingly added to those lists. If you recently installed or tightened a security plugin and then noticed AI products claiming they cannot read your pages, this is a common cause.
Origin server configuration
Nginx, Apache, Caddy and similar servers can deny based on $http_user_agent. This is reliable when you control the origin and traffic reaches it. If a CDN sits in front, the CDN’s decision usually happens first.
Practical approach to a hard block
- Decide the policy — which named crawlers should be blocked at the edge, and which should still be allowed if robots.txt permits them.
- Implement the rule in the system that actually terminates the connection (CDN first if you use one).
- Keep robots.txt consistent so compliant agents do not waste requests and so your public policy matches enforcement.
- Verify with an external test — change configuration, then run Movio’s checker or an equivalent tool from outside your network. Confirm that the server result shows the expected status code for each crawler.
False positives and over-blocking
Aggressive bot rules can also affect:
- Monitoring and uptime checkers that use unusual User-Agents.
- Research crawlers and academic projects.
- Some legitimate AI product features that fetch pages when a user explicitly asks for them.
If users report that “ChatGPT cannot open my page” after you enable a broad AI block, that may be intended. If monitoring tools start failing, refine the rule to target specific User-Agents rather than a vague “AI” category.
What Movio can and cannot see
Movio sends ordinary HTTP requests with documented AI User-Agent strings from its own servers. It reports status codes, redirects, timing, and some indicators of challenge pages or known CDN fingerprints. It does not:
- Solve CAPTCHAs or JavaScript challenges.
- Spoof the operator’s published IP ranges.
- Reproduce TLS fingerprints or browser-like behaviour beyond the User-Agent header.
If a site only allows traffic from a crawler’s verified IP list, Movio’s result may differ from the real crawler’s experience. The report labels uncertainty rather than pretending the test is complete.
Combining layers
A durable setup for most sites looks like this:
- robots.txt expresses the public policy for compliant agents.
- CDN/WAF rules enforce hard blocks for the agents you never want to serve.
- Periodic external checks confirm that both layers still match the intended policy after plugin updates, CDN rule changes, or hosting migrations.
For the policy question of whether blocking is the right choice at all, see Should you allow AI crawlers on your website?.