How Movio's AI Crawler Firewall Checker Works
The checker makes a series of real HTTP requests from the Movio.site server to the page you enter. Nothing is simulated. Here is what happens, in order.
- URL validation. The address must use http or https on the standard ports. Movio rejects addresses with embedded passwords, raw IP addresses, internal host names and anything that resolves to a private, loopback, link-local or otherwise non-public network. This protects other people's networks and Movio's own.
- robots.txt retrieval. Movio requests
/robots.txtfrom the same site and records the status code, any redirects, the file size and whether the content looks like a real robots.txt file or, for example, an HTML error page. - robots.txt rule analysis. For each crawler, the rules are read the way the robots exclusion standard (RFC 9309) describes: a group naming the crawler wins over the
*group, the longest matching path rule wins, and Allow wins a tie. The rule that decided the outcome is shown in the report. - Crawler-specific HTTP requests. Movio sends one GET request per crawler using that crawler's documented User-Agent string, plus one neutral control request that does not claim to be an AI crawler. The control request is what lets the report tell "this site rejects everything automated" apart from "this site rejects this crawler".
- Server response analysis. Each response is recorded: status code, redirect chain, final URL, response time, content type and size, and a short list of harmless headers. Responses are read only up to a size limit.
- WAF and challenge detection. Movio looks for recognisable indicators: challenge-page markup, CAPTCHA widgets on error pages, block-page wording, and headers or cookies that reveal a protection service. These are labelled as possible evidence. A service being visible in the path is reported separately from a challenge actually being served.
- Final result. Robots.txt evidence and server evidence are combined into one status per crawler (Accessible, Blocked, Warning or Unknown) with a confidence level and a plain-language reason.
Movio does not solve CAPTCHAs, bypass authentication, rotate IP addresses or imitate browser fingerprints. It only observes how a server answers ordinary requests.
How to read your results
Statuses
- Accessible: robots.txt does not disallow the crawler and the server answered the test request normally.
- Blocked: robots.txt disallows the crawler, or the server rejected the request in a way that points specifically at that crawler.
- Warning: something needs attention, such as a challenge page, a rate limit, or a rejection that the neutral control request also received.
- Unknown: the test could not support a conclusion, for instance a timeout or a connection error.
Confidence
- Confirmed: directly read from robots.txt or directly observed in the response.
- Likely: strongly suggested, for example a 403 sent to the crawler while the control request succeeded.
- Possible: a weak or ambiguous signal that deserves a look.
- Unknown: not enough evidence either way.
The status describes the test request Movio sent. It says "this request was rejected", not "the real crawler will be rejected".
Robots.txt vs Firewall Blocking
Two separate mechanisms decide whether a crawler gets your page, and they do not know about each other.
robots.txt is a set of instructions
It is a public text file at /robots.txt. It tells crawlers which parts of a site you would like them to fetch or avoid. A well-behaved crawler reads it first and follows it. Nothing enforces it: a client that never reads the file can still request every page.
A firewall is server-side access control
A firewall, WAF, CDN rule, hosting security setting or security plugin looks at each incoming request and decides whether to answer it. It can use the User-Agent, IP address, request rate, country or behaviour. It enforces its decision whatever robots.txt says.
That is why a site can have this in robots.txt:
User-agent: GPTBot
Allow: /and still return HTTP 403 Forbidden to a request that identifies as GPTBot, because a security rule rejects it before robots.txt is ever relevant. The reverse also happens: the server happily serves the page, but robots.txt says Disallow: /, so a compliant crawler stays away.
When both layers say yes, a crawler can read the page. When either says no, it cannot. Diagnosing a problem means checking both layers, which is what the report's two columns are for.
Crawlers Movio tests
The list below shows how each name is tested. Operators can change User-Agent strings and behaviour at any time, so treat this as a snapshot and confirm details in each operator's own documentation.
| Crawler | Operator | What the operator describes | How Movio tests it |
|---|---|---|---|
| GPTBot | OpenAI | Crawler the operator describes as collecting content that may be used to train generative AI models. | robots.txt rule and a GET request with its User-Agent |
| OAI-SearchBot | OpenAI | Crawler the operator describes as used for search features in its assistant. | robots.txt rule and a GET request with its User-Agent |
| ChatGPT-User | OpenAI | Fetcher used when a person asks the assistant to open a page. Check the operator documentation for how it treats robots.txt. | robots.txt rule and a GET request with its User-Agent |
| ClaudeBot | Anthropic | Crawler the operator describes as collecting public web content for model development. | robots.txt rule and a GET request with its User-Agent |
| anthropic-ai | Anthropic | Older robots.txt token. No separate crawl request is documented under this name, so only the robots.txt rule is read. | robots.txt rule only (no server request is made under this name) |
| Google-Extended | robots.txt control token for certain Google AI uses. It does not make requests of its own, so only the robots.txt rule is read. | robots.txt rule only (no server request is made under this name) | |
| GoogleOther | Generic Google crawler used by product teams for fetching public content. | robots.txt rule and a GET request with its User-Agent | |
| PerplexityBot | Perplexity | Crawler the operator describes as building the index behind its answer engine. | robots.txt rule and a GET request with its User-Agent |
| Amazonbot | Amazon | Crawler the operator describes as improving its services, including assistant answers. | robots.txt rule and a GET request with its User-Agent |
| Applebot-Extended | Apple | robots.txt control token governing certain AI uses of content Apple already crawled. It does not crawl, so only the robots.txt rule is read. | robots.txt rule only (no server request is made under this name) |
| Bytespider | ByteDance | Crawler operated by ByteDance. | robots.txt rule and a GET request with its User-Agent |
| Meta-ExternalAgent | Meta | Crawler the operator describes as collecting content for AI model training and indexing. | robots.txt rule and a GET request with its User-Agent |
| Meta-ExternalFetcher | Meta | Fetcher used for user-initiated requests. Check the operator documentation for how it treats robots.txt. | robots.txt rule and a GET request with its User-Agent |
| CCBot (Common Crawl) | Common Crawl | Crawler for the open Common Crawl dataset. "CCBot" and "Common Crawl" are the same crawler, so they are tested as one entry. | robots.txt rule and a GET request with its User-Agent |
| cohere-ai | Cohere | Token used in robots.txt. No full User-Agent string is publicly documented, so the server test sends only the token and is an approximation. | robots.txt rule and a GET request with its User-Agent |
Movio does not use these names as a brand and has no relationship with any of the operators.
Why AI crawler access matters
More people now get information through AI-powered search and assistants that read web pages, either ahead of time to build an index or on demand while answering a question. For a website owner, the first practical question is simple: can those crawlers reach my pages at all?
Often the answer is decided by accident. A robots.txt file written years ago, a security plugin installed with strict defaults, or a CDN feature that blocks unfamiliar bots can all shut crawlers out without anyone choosing to. Equally, many owners choose to block some crawlers on purpose, for example to keep content out of model training, and want to confirm the block actually works.
Either way, the useful thing is to know what is really happening. Movio helps you see which layer is making the decision. It does not tell you whether you should allow any crawler. That depends on your content, your business and your views on how your work is used.
Allowing a crawler removes a barrier; it does not promise anything in return. Whether an AI product reads, cites or links to your pages, and whether that sends you visitors, is up to its operator.
Troubleshooting common results
robots.txt allows the crawler, but the server returns 403
A security layer is rejecting the request. Check, in roughly this order: the CDN or WAF dashboard (look for bot-management and managed rules, and for one-click settings that block known AI bots), hosting-level firewalls such as ModSecurity rules, WordPress security plugins, and rate-limit or country rules. Event logs usually show the rule that matched.
A challenge page was detected
Crawlers cannot complete CAPTCHAs or JavaScript challenges. If you want a crawler to read the page, add a rule that lets it through, ideally one that verifies the crawler by the operator's published IP ranges rather than by User-Agent alone, since User-Agent strings are trivial to fake.
The control request failed too
If the neutral request got the same response, the site is rejecting automated clients in general, or the page requires a login or is missing. Test a public page such as your home page first.
HTTP 429
The server is rate limiting. Movio sends one request per crawler in a short burst, which some sites treat as heavy. Wait a few minutes and test again.
Timeouts
A timeout is reported as Unknown. Test again, check server load, and look for security rules that silently drop connections instead of answering.
robots.txt is missing, returns an error, or looks like HTML
A missing file (404) means no restrictions. A server error (5xx) makes crawlers uncertain and some pause crawling. A robots.txt that returns an HTML page contains no valid rules. Fix these so the file returns plain text with HTTP 200.
Results differ from what you expect for a Google or Apple name
Google-Extended and Applebot-Extended are robots.txt tokens only. They are read from robots.txt, and no server test applies.
Important Limitations
A remote HTTP test cannot perfectly reproduce every proprietary bot-detection system. Websites may decide based on many signals Movio cannot copy, including:
- IP reputation and the network (ASN) a request comes from
- Allow-lists of crawler IP ranges published by the operator
- TLS fingerprints and HTTP/2 characteristics
- Browser fingerprints and JavaScript challenges
- Behavioural detection across many requests
- Geolocation rules
- Advanced or custom WAF rules
Because of this, Movio can provide a diagnostic signal, but it cannot guarantee exactly how an AI company's own crawler infrastructure will behave. A page marked Accessible may still be blocked for the real crawler, and a page marked Blocked may be open to it. Use the report to find and fix configuration problems, and check your server logs for real crawler traffic when accuracy matters.
Movio also tests one URL, not your whole site, and only sends one request per crawler, so rules that trigger after repeated requests will not appear.
Frequently asked questions
What is an AI crawler?
An AI crawler is an automated program that requests web pages on behalf of an AI company. Depending on the crawler, the pages may be used to build a search index, to answer a question in real time, or to collect material for model development. Each crawler identifies itself with a name in the User-Agent header, and most operators publish documentation describing what that crawler does.
What is GPTBot?
GPTBot is the crawler OpenAI documents for collecting web content that may be used to train its generative AI models. You can allow or disallow it in robots.txt using the token GPTBot. It is separate from OAI-SearchBot and ChatGPT-User, which OpenAI documents for other purposes.
What is OAI-SearchBot?
OAI-SearchBot is an OpenAI crawler described as supporting search features in its assistant. Because it has its own robots.txt token, a site can treat it differently from GPTBot. Check OpenAI's current documentation for the exact behaviour, as operators change these details over time.
What is ClaudeBot?
ClaudeBot is the crawler Anthropic documents for collecting public web content for model development. It follows robots.txt rules written for the token ClaudeBot. Movio tests it with a request that carries the ClaudeBot User-Agent string.
What is PerplexityBot?
PerplexityBot is the crawler Perplexity documents for building the search index behind its answer engine. Like the others it can be allowed or disallowed with a robots.txt group named PerplexityBot, and a firewall can still reject it independently.
Can robots.txt block AI crawlers?
Yes. A User-agent group naming the crawler, followed by Disallow: /, tells a compliant crawler not to fetch your pages. robots.txt is a request, not an enforcement mechanism: it works when the crawler's operator chooses to follow it. It cannot stop a client that ignores it.
Can a firewall block AI crawlers?
Yes. A firewall, WAF, CDN rule or bot-protection service decides on the server side whether a request is answered. It can reject requests by User-Agent, IP address, rate, geography or behaviour. This works regardless of what robots.txt says.
Why does robots.txt say Allow but the checker shows 403?
They are separate systems. robots.txt tells crawlers what you permit, while a firewall or security plugin independently decides whether to serve the request. If a security rule rejects the crawler's User-Agent or IP, the server returns 403 even though robots.txt allows it. The fix is in the security layer, not in robots.txt.
What does HTTP 403 mean?
HTTP 403 Forbidden means the server understood the request and refused to fulfil it. It is commonly produced by firewalls, bot protection, hosting security rules and permission settings. By itself, a 403 does not say which rule caused it.
Can Movio detect Cloudflare?
Movio can notice Cloudflare indicators in a response, such as certain headers, cookies and the markup of a Cloudflare challenge page. It reports them as possible evidence. Seeing Cloudflare in the request path does not mean Cloudflare blocked anything, so the report separates "present" from "challenge detected".
Can Movio bypass a firewall?
No, and it is not designed to. Movio sends ordinary requests with a crawler User-Agent and reports what comes back. It does not solve CAPTCHAs, evade rate limits, forge IP addresses or imitate security fingerprints.
Does Movio store my website URL?
Not permanently. While a check runs, and for a short time afterwards so that results can be cached and downloaded, the address and results are held in temporary files that expire automatically. Normal web server logs may still record that a request was made. See the Privacy Policy for details.
Can this checker guarantee AI crawler access?
No. The test runs from Movio's server, not from the crawler operator's infrastructure. Sites that use IP reputation, verified crawler IP ranges, TLS fingerprints or JavaScript challenges may treat the real crawler differently. The report is a diagnostic signal, not a guarantee.
Why are Google-Extended and Applebot-Extended not tested with a server request?
They are robots.txt control tokens. The operators do not send requests under those names, so there is no User-Agent to test against a firewall. Movio reads the robots.txt rule for them and states that no server test applies.
What does a timeout mean in the report?
It means the server did not answer within the test window. That can come from a slow server, network trouble or a security rule that drops requests, but a timeout alone does not show which. Movio reports it as Unknown rather than as a block.
Does allowing an AI crawler bring me traffic or better visibility?
No result is promised. Allowing a crawler only removes a barrier to it reading your pages. Whether any AI product uses your content, cites it or sends visitors is decided by the operator and is outside your control and Movio's.