Cloudflare's robots.txt files can now say three different things at once. Since September 24, 2025, the company has let site owners set separate values for search indexing, AI retrieval at query time, and AI model training inside a single line, rather than the old all-or-nothing disallow rule. More than 3.8 million domains already use Cloudflare's managed robots.txt to express some version of that preference. Most of them still set one blanket rule for all three, though, which means a brand trying to stop AI training often ends up blocking the very citations it wants from ChatGPT or Perplexity.
The three permissions one crawl rule used to control
A disallow line in a traditional robots.txt has only ever meant one thing: stay out, entirely. Cloudflare's Content-Signal syntax splits that single permission into three: whether a crawler can index a page for search, use it to ground a live answer, and train a model on it, all set independently in the same file.
The syntax looks like this: Content-Signal: search=yes, ai-input=yes, ai-train=no, placed under a user-agent block the same way an Allow or Disallow line would be. The middle value, ai-input, is the one that actually matters for citations: it governs whether a crawler can pull a page into a retrieval-augmented generation system and use it to ground the answer a user reads in ChatGPT, Perplexity, or Gemini right now. The ai-train value governs something unrelated: whether that same crawl feeds a training run that might reshape the model's weights months later. A brand can set ai-train=no and ai-input=yes on the same page, refusing to become training data while still allowing the content to be cited today. Before Content-Signal existed, robots.txt had no way to express that difference. A disallow line blocked everything a given crawler might do with a page, so a publisher who wanted to stop training was, by construction, also blocking retrieval.
AI input and AI training are not the same permission. A page can ground a real-time answer in ChatGPT or Perplexity today while still refusing to feed a model's next training run, but only if the signal it uses actually keeps those two permissions separate.
Where robots.txt still falls short
Robots.txt only reaches crawlers that fetch the file, parse it correctly, and choose to honor it, and even a fully honored rule applies to a whole user agent rather than to one page. Which AI bots actually keep that promise is its own question; the gaps left over are what pushed publishers toward page-level and header-level signals instead.
OpenAI's own crawler documentation is unusually direct about this. Disallowing GPTBot, it states, indicates a site's content should not be used in training generative AI foundation models, and OpenAI says it honors that rule. But the same documentation carves out an exception for ChatGPT-User, the crawler that fetches a page on a person's behalf inside a live chat: because these actions are initiated by a user, robots.txt rules may not apply. A page a site disallowed for training can still be fetched and shown to that user seconds later. Robots.txt also can't separate a page written for search snippets from one written to answer a specific question, since the rule applies to the whole site or the whole user agent, never to a single URL's purpose.
Meta tags and HTTP headers reach individual pages
A robots meta tag in a page's head, or the equivalent X-Robots-Tag in its HTTP response, controls indexing and serving for that one URL instead of an entire site, according to Google's own documentation. That page-level reach is why publishers layer meta tags and headers on top of robots.txt rather than replacing it.
The best-known example is the noai and noimageai pair of directives, which DeviantArt introduced in November 2022 to keep AI image generators from training on artists' uploads. Adoption has grown well beyond that original use case: Originality.AI's ongoing tracker counted more than 88,000 domains carrying one of the two tags by June 2026, split between 77,645 using the meta tag and 23,604 using the header, a 26.5% jump in meta-tag adoption in a single month. None of that adoption comes with a guarantee. A noai tag is a request, not a lock, Originality.AI notes, since no major AI lab has published a policy committing to honor it. Google's own robots meta tag carries an AI-relevant side effect that's easy to miss, too: the nosnippet directive, built years before generative answers existed, now also keeps a page from being used as direct input for AI Overviews and AI Mode.
TDMRep adds a legal hook the others don't have
The Text and Data Mining Reservation Protocol, or TDMRep, is a W3C Community Group standard, finalized May 10, 2024, for declaring mining rights in a machine-readable way. Publishers set it through an HTTP header, an HTML meta tag, or a single sitewide tdmrep.json file, and it can point to a licensing policy rather than a flat yes or no.
That structure sets TDMRep apart from robots.txt and the noai tag, both of which are purely voluntary wherever they're read. A publisher reserves mining rights with a header as simple as tdm-reservation: 1, or sets it to 0 to signal unrestricted mining, and can attach a tdm-policy header pointing to a licensing page for anyone who wants to negotiate access anyway. European Union copyright rules give rightsholders an explicit text-and-data-mining opt-out when it's expressed in a machine-readable way, and TDMRep was built to be exactly that expression. Outside the EU, the protocol still depends entirely on a crawler operator choosing to check for it and comply, the same limitation every signal in this piece shares.
No signal here is actually enforced
None of these mechanisms, from robots.txt to Content-Signal to TDMRep, comes with a technical guarantee. Each depends on the crawler operator voluntarily reading the file and honoring it, which is why the International Press Telecommunications Council's 2025 guidance tells publishers to layer several signals rather than trust any single one.
Robots.txt is, in the IPTC's words, seemingly the simplest method but actually one of the most difficult to get right, since every AI crawler's user agent has to be named and blocked separately, and the list keeps growing. OpenAI does publish a clear policy for one of its four crawlers, saying a disallow on GPTBot keeps that content out of its foundation-model training data, and that it honors the rule. No equivalent public commitment exists yet from OpenAI, Anthropic, or Google for the Content-Signal or noai layers specifically, which is exactly why the IPTC's guidance treats robots.txt, meta tags, HTTP headers, and tdmrep.json as complementary rather than as substitutes for each other.
What to set if the goal is citations, not silence
A brand that wants to appear in ChatGPT and Perplexity answers without feeding a future training run has one clear move: set ai-input=yes and ai-train=no in Cloudflare's Content-Signal line, and pair it with an explicit crawler allow in robots.txt, rather than a blanket noai tag that most systems read as leave this alone entirely.
Setting the signal correctly is only half the job, since nothing described here is enforced by anything other than the crawler operator's own choice. Server logs are the only way to confirm whether GPTBot, PerplexityBot, or Claude-SearchBot are actually fetching the pages a site meant to leave open, and a closer look at analyzing that traffic shows how to tell a real crawl from a spoofed user agent. Cloudflare's own bot-management defaults have separately been shown to block legitimate AI crawlers even when a site's Content-Signal line says otherwise, which is worth checking directly rather than assuming the two settings agree with each other.
Frequently Asked Questions
Does robots.txt actually block AI training?
It can, but only for crawlers that choose to honor it. OpenAI's own documentation says disallowing GPTBot in robots.txt means a site's content shouldn't be used to train its foundation models, and the company states it respects that rule. Robots.txt has no way to separately allow the same content to be used for retrieval or citation, though, which is what Cloudflare's Content-Signal syntax was built to fix.
What is the noai meta tag, and does ChatGPT respect it?
Noai and noimageai are meta tags DeviantArt introduced in November 2022 to discourage AI training on uploaded artwork. More than 88,000 domains now use one of the two tags, according to Originality.AI, but no major AI company, including OpenAI, has published a policy committing to honor them. They function as a public request, not a technical block.
What is Cloudflare's Content Signals Policy?
It's a robots.txt extension Cloudflare launched on September 24, 2025 that lets a site set three separate values, search, ai-input, and ai-train, instead of one blanket allow or disallow rule. More than 3.8 million domains already use Cloudflare's managed robots.txt to express some version of this preference.
How is TDMRep different from robots.txt?
TDMRep is a W3C standard, finalized in May 2024, that publishers set through an HTTP header, a meta tag, or a sitewide tdmrep.json file to declare text-and-data-mining rights. Unlike robots.txt, it can carry legal weight under EU copyright rules, though outside the EU it still depends on the crawler choosing to check for it.
Can a brand block AI training but still get cited by ChatGPT or Perplexity?
Yes, in principle. Setting ai-input=yes and ai-train=no in Cloudflare's Content-Signal line separates the two permissions, letting a crawler ground a live answer without feeding a training run. A blanket noai tag doesn't make that distinction, so brands that want citations should avoid it and confirm the split is working with server log analysis.