Control token · Apple
Applebot-Extended
A robots.txt token that controls whether pages Applebot fetched may be used to train Apple's foundation models. Disallowing it does not remove a site from Siri or Spotlight suggestions.
- Operator
- Apple
- Purpose
- Robots.txt control tokens
- Feeds
- Apple Intelligence
- robots.txt token
Applebot-Extended- Honours robots.txt
- Honours robots.txt
- Documentation
- support.apple.com
Does it appear in server logs?
No. Applebot-Extended is a token the operator reads in robots.txt, not a user agent that fetches pages. You will never see a request carrying it; the pages are fetched by the operator's ordinary crawler. The string below is what a tool would match if one were ever sent.
Mozilla/5.0 (compatible; Applebot-Extended/1.0; +http://www.apple.com/go/applebot)How to block Applebot-Extended
Add this to the robots.txt at the root of your domain. Robots.txt is advisory: it works only on crawlers that choose to read it. This one is honours robots.txt according to its operator.
User-agent: Applebot-Extended
Disallow: /
To block every robots.txt control token at once, or to block training while keeping AI search citations, use the robots.txt generator.
How to verify a genuine hit
The user-agent string is self-asserted: anyone can send it from a laptop, and scrapers routinely impersonate well-known crawlers to slip past filters aimed at unknown ones.
No request carries this token; pages are fetched by Applebot, whose hits can be verified by reverse DNS to *.applebot.apple.com.