What are AI crawlers and how do they work?
AI crawlers are automated programs that fetch web pages on behalf of AI companies and products. Some collect text to train or update large language models. Others retrieve pages in real time so an assistant can answer a question with current information. A third group focuses on citations and grounding — showing users where an answer came from.
They share infrastructure patterns with traditional search-engine crawlers, but their goals, rate limits, User-Agent strings and policy expectations are different. Understanding those differences is the first step toward deciding how your site should respond to them.
How AI crawlers differ from search crawlers
Search crawlers (Googlebot, Bingbot and similar) primarily build and refresh an index used for ranking and displaying search results. Their success is measured by how well they discover and keep up-to-date pages that users might search for.
AI crawlers often have additional or different purposes:
- Training / pre-training data — collecting large volumes of text to improve a model’s knowledge and language ability.
- Retrieval-augmented generation (RAG) — fetching specific pages at query time so the model can ground an answer in current content.
- Citation and attribution — recording source URLs so products can show links alongside generated answers.
- Product features — powering “browse” modes, shopping assistants, research tools and similar experiences.
Because the purpose differs, the same robots.txt or firewall decision can have different consequences for search visibility versus AI visibility. Blocking an AI crawler does not automatically affect Google or Bing rankings, and allowing it does not guarantee citations or traffic.
Common AI crawlers site owners encounter
Names and behaviour change over time. The list below reflects the crawlers that Movio tests and that appear frequently in public documentation and logs as of late 2026. Always confirm details in the operator’s own documentation.
- GPTBot / ChatGPT-User — associated with OpenAI. GPTBot is commonly linked to training and indexing; ChatGPT-User is often used when a user asks the product to fetch a page.
- ClaudeBot / Claude-User — associated with Anthropic. Similar distinction between background crawling and user-initiated fetches.
- Google-Extended — a robots.txt token that controls whether Google may use content for certain Gemini / AI features. It is not a separate User-Agent that makes HTTP requests under that name in the same way GPTBot does.
- PerplexityBot — used by Perplexity for its answer engine and related products.
- Applebot-Extended — Apple’s token for AI-related use of content, again primarily controlled via robots.txt.
- Others — Amazonbot, Bytespider, CCBot (Common Crawl), Cohere and additional product-specific agents appear regularly in logs.
Some operators publish IP ranges or verification methods so site owners can confirm a request really comes from them. Others rely mainly on User-Agent strings. Movio’s reports treat User-Agent based tests as diagnostic signals, not cryptographic proof of identity.
The two layers that decide access
Whether an AI crawler can read a page is almost always decided by two independent layers:
- robots.txt — a voluntary protocol. Compliant crawlers read the file and avoid paths they are told not to fetch. Non-compliant crawlers may ignore it.
- Server / network layer — firewalls, CDNs, WAFs, bot-management products and application security plugins. These can return 403, challenges, or drop connections regardless of what robots.txt says.
A common surprise is a site whose robots.txt allows a crawler while a security rule still blocks it — or the reverse. Movio’s checker is built specifically to show both results side by side. See the robots.txt guide and the firewall guide for practical configuration.
Why site owners started paying attention
Until the mid-2020s, many sites treated “bots” as a single category: allow search engines, block everything else that looked aggressive. The rise of generative AI changed the stakes. Publishers began to ask whether their content was being used to train models that competed with their own products, whether AI answers were substituting for visits, and whether selective blocking was technically feasible.
At the same time, some AI products began showing citations and sending referral traffic. The result is a fragmented landscape: some sites block most AI crawlers, some allow them selectively, and some leave defaults in place. There is no single industry consensus, which is why diagnostic tools and clear documentation matter more than one-size-fits-all advice.
What Movio does and does not claim
Movio reports what happens when a request that presents a known AI User-Agent reaches your site from Movio’s servers, and what your robots.txt says about that agent. It does not claim that the real operator’s infrastructure will always behave the same way, and it does not recommend a policy. For a decision framework rather than a technical checklist, see Should you allow AI crawlers on your website?.