OpenAI runs four separate crawlers under four separate names. Anthropic runs three. Perplexity runs two. Disallowing one does not disallow the others. A single line reading "Disallow: GPTBot" in robots.txt stops OpenAI's training crawler and nothing else: not ChatGPT's search bot, not its ad-verification bot, not the fetcher that grabs a page the moment someone pastes a link into a chat. Plenty of marketers set up their AI crawler rules once, back in 2024, and never revisited them. Here is what each engine's crawlers actually do, and what robots.txt can and cannot control in 2026.
Why one robots.txt rule can't "block AI"
robots.txt controls crawlers one user-agent at a time, and every major AI company now operates more than one. Blocking a lab's flagship bot only stops the function that bot performs, usually training data collection, while its search-indexing bot and its live user-request bot keep fetching pages under entirely different names.
The mix has also shifted underneath site owners. Cloudflare's own network data put AI training-crawler requests at 52% of all crawler traffic in June 2026, up from 22% a year earlier, and non-human requests crossed 50% of everything Cloudflare sees for the first time. That single number explains why the old instinct, one blanket "block all AI" rule, stopped making sense: most of the traffic hitting that rule today isn't training data collection at all.
What GPTBot, OAI-SearchBot, and ChatGPT-User each actually do
OpenAI documents four separate crawlers, each controllable on its own in robots.txt. GPTBot collects data for training OpenAI's foundation models. OAI-SearchBot indexes pages specifically for ChatGPT's search feature. OAI-AdsBot checks landing pages submitted with ads. ChatGPT-User fetches one specific page only when a person asks ChatGPT to open it.
Disallowing GPTBot does not remove a site from ChatGPT's search answers. That requires a separate rule for OAI-SearchBot, according to OpenAI's own developer documentation. And ChatGPT-User isn't really a crawler in the traditional sense: OpenAI's docs note it handles live "user actions," and changes to its access can take roughly 24 hours to propagate once a site updates its rules. We've covered how ChatGPT actually chooses which sources to surface in more depth, including what happens after a page clears the crawl stage.
Anthropic split ClaudeBot into three bots with three different jobs
Anthropic reorganized its crawler documentation in February 2026, breaking what used to be a single ClaudeBot listing into three named agents. ClaudeBot still collects data for model training. Claude-User fetches a page the moment someone asks Claude a question about it. Claude-SearchBot indexes content specifically for Claude's search-style answers, and disallowing it separately can hurt a brand's visibility there even while ClaudeBot itself stays blocked.
That's a reversal of the 2024 default, when most sites either blocked "ClaudeBot" outright or left it wide open with no in-between option. Search Engine Journal, covering the February 2026 update, noted the change makes robots.txt decisions "more granular," which in practice means a site can now refuse Anthropic's training data while still keeping its content eligible for Claude's live answers. Our Claude citation playbook walks through what to allow if citations, not training exclusion, are the actual goal.
PerplexityBot follows robots.txt. Perplexity-User doesn't have to
Perplexity operates two crawlers with opposite obligations. PerplexityBot, used to build its search index, follows robots.txt, and Perplexity's own documentation asks sites to allow it rather than block it. Perplexity-User, triggered when someone asks Perplexity a direct question about a page, isn't bound by the same rule, because the fetch stands in for a human's own request rather than bulk indexing.
A robots.txt rule is only as good as the crawler's willingness to identify itself honestly. Not every company treats that as a given.
In August 2025, Cloudflare publicly accused Perplexity of using an undisclosed, Chrome-spoofing crawler to keep pulling pages from sites that had already disallowed PerplexityBot. Cloudflare CEO Matthew Prince wrote that some "AI companies act more like North Korean hackers" than legitimate crawlers. Perplexity first denied the bot was theirs, then said the traffic came from a third-party service, and argued that user-driven fetching is fundamentally different from automated crawling. The dispute is over a year old now, but it's why IP verification, not just user-agent strings, has become standard advice; OpenAI, for one, publishes its crawler IP ranges directly at openai.com/gptbot.json for exactly this reason. Our guide to Cloudflare quietly blocking AI crawlers and our Perplexity optimization guide both dig into the mechanics on the Perplexity side specifically.
Google-Extended blocks AI training, not your Google rankings
Google-Extended is a separate product token from Googlebot, and disallowing it only removes a site from training and grounding data for Gemini and Vertex AI. Google's own documentation, updated in April 2025, states plainly that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
That distinction matters because it's the opposite of how most site owners assume robots.txt works. Blocking Google-Extended costs a site nothing in organic Search rankings; it only trades away potential visibility inside Gemini answers and AI Overviews that draw on Google's training and grounding pipeline. Sites weighing that trade should read our guide to optimizing for Gemini before deciding either way.
robots.txt is becoming a negotiating table, not just a wall
Cloudflare will begin blocking "mixed-use" AI crawlers by default on ad-supported pages starting September 15, 2026. The policy forces AI companies to separate their training, search, and agent crawlers into distinguishable identities, or lose default access on any Cloudflare-protected site that hasn't explicitly allowed them. The rule applies to new Cloudflare customers, new sites on existing accounts, and every free-tier site.
Alongside the blocking default, Cloudflare introduced Pay Per Crawl and Pay Per Use, letting sites charge AI companies for access instead of only allowing or denying it outright. robots.txt still decides who gets through the door first, but for a growing number of sites it's no longer the only lever. Two mechanisms are easy to confuse here: robots.txt governs whether a bot may fetch a page at all, while llms.txt is a separate, unenforced file that suggests how an engine should use a page once it's already been fetched. Getting the access rules right is still only half the job. Tools like VizibleAI track whether opening the door to a crawler actually increases how often a brand gets cited across ChatGPT, Claude, Gemini, and Perplexity, a number access rules alone can't show. Pairing a crawler audit with a proper GEO audit is the more complete way to see the effect.
Frequently Asked Questions
Does Disallow: GPTBot stop my content from appearing in ChatGPT answers?
Not entirely. GPTBot only governs training data collection. ChatGPT can still surface your pages through OAI-SearchBot, its dedicated search crawler, and through ChatGPT-User when someone pastes your URL directly into a chat. To stop appearing in ChatGPT's search results specifically, you need a separate Disallow rule for OAI-SearchBot, per OpenAI's own crawler documentation.
Do I need to block Claude-User if I already block ClaudeBot?
No, and most sites shouldn't. ClaudeBot handles training data collection, while Claude-User fetches a single page only when a person asks Claude about it directly. Blocking Claude-User removes your site from Claude's ability to answer specific user questions about it, which is usually the opposite of what a brand chasing AI visibility wants.
Does blocking Google-Extended hurt my Google Search rankings?
No. Google's own documentation states explicitly that Google-Extended isn't a ranking signal and doesn't affect inclusion in Google Search. It only controls whether your content trains or grounds Gemini and Vertex AI models. Googlebot, which handles Search indexing, is a separate user-agent entirely unaffected by Google-Extended rules.
How do I verify a bot claiming to be PerplexityBot is actually PerplexityBot?
Check the requesting IP address against the ranges each company publishes, rather than trusting the user-agent string alone; OpenAI, for instance, posts its IP list at openai.com/gptbot.json. After Cloudflare's August 2025 accusation that an undisclosed crawler impersonated a browser to bypass Perplexity's own robots.txt rules, IP verification became standard advice rather than an edge case.
Should I just block every AI crawler to protect my content?
That depends on the goal. Blocking training crawlers like GPTBot or ClaudeBot keeps content out of model training, but it doesn't remove a brand from AI answers, since separate search and user-fetch bots keep operating under different names. Sites optimizing for AI visibility generally allow the search and user-facing bots while making a case-by-case call on training bots.
Will robots.txt still matter after Cloudflare's pay-per-crawl rollout?
Yes, it remains the first gate. Cloudflare's September 2026 policy blocks undifferentiated "mixed-use" crawlers by default on ad-supported pages, but sites still need robots.txt rules to state which named bots they permit. The pay-per-crawl tools sit on top of that as a monetization layer, not a replacement for it.
Turning crawler access into actual AI visibility
Getting the robots.txt rules right only controls who can fetch your pages. It says nothing about whether ChatGPT, Claude, Gemini, or Perplexity actually cite your brand once they do. VizibleAI tracks brand mentions, position, and share of voice across those engines, so a brand can see whether opening the door to a crawler produced an actual citation.