On demand · Meta
Meta-ExternalFetcher
Fetches a specific link a person shared with Meta AI. Meta documents that because the request is user-initiated it may bypass robots.txt rules.
- Operator
- Meta
- Purpose
- User-triggered fetchers
- Feeds
- Meta AI (link fetch)
- robots.txt token
Meta-ExternalFetcher- Honours robots.txt
- May ignore robots.txt
- Documentation
- developers.facebook.com
What it looks like in server logs
A representative user-agent string. Version numbers change; the Meta-ExternalFetcher token does not. Paste your own log line into the user-agent checker to confirm a match.
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)How to block Meta-ExternalFetcher
Add this to the robots.txt at the root of your domain. Robots.txt is advisory: it works only on crawlers that choose to read it. This one is may ignore robots.txt. To enforce a block, match the user agent at your CDN or web server and return 403.
User-agent: Meta-ExternalFetcher
Disallow: /
To block every user-triggered fetcher at once, or to block training while keeping AI search citations, use the robots.txt generator.
How to allow Meta-ExternalFetcher while blocking others
A more specific User-agent group wins over a wildcard. Place this above any User-agent: * block that disallows paths, and Meta-ExternalFetcher keeps full access.
User-agent: Meta-ExternalFetcher
Allow: /
How to verify a genuine hit
The user-agent string is self-asserted: anyone can send it from a laptop, and scrapers routinely impersonate well-known crawlers to slip past filters aimed at unknown ones.
Meta publishes its crawler IP ranges via its AS32934 route announcements.