Applebot-Extended

Applebot-Extended is Apple's opt-out token, used in robots.txt, that lets a site exclude its content from training Apple Intelligence and Apple's other generative AI features, separate from the standard Applebot rules governing Apple's search and Siri-related indexing across the web.

Why it matters

As Apple Intelligence expands its role in Siri and on-device search, Applebot-Extended is a rule worth deliberately setting rather than leaving unconfigured, since an unconfigured directive defaults to allowing the training use rather than opting a site out of it.

Related terms

Google-Extended

Google-Extended is not a crawler itself, but a robots.txt token site owners use to opt specific content out of use by Gemini and Google's other generative AI products, separate from the standard Googlebot rules governing traditional search indexing. A page can stay fully indexed in Google Search while excluded from Gemini's training data.

robots.txt (AI directives)

robots.txt is the long-standing standard crawler-control file at a site's root, now extended with AI-specific user-agent rules that separately allow or disallow crawlers like GPTBot, ClaudeBot, and PerplexityBot. A single robots.txt file can permit standard search crawling while blocking one or more AI crawlers, or the reverse.

Optimize your presence across Generative AI and Answer Engines.

Track your AI visibility · Get a demo