Control token · Webz.io
Webzio-Extended
A robots.txt token that lets a site stay in Webz.io's general dataset while opting out of the subset sold for AI training.
- Operator
- Webz.io
- Purpose
- Robots.txt control tokens
- Feeds
- Webz.io corpus
- robots.txt token
Webzio-Extended- Honours robots.txt
- Honours robots.txt
- Documentation
- webz.io
Does it appear in server logs?
No. Webzio-Extended is a token the operator reads in robots.txt, not a user agent that fetches pages. You will never see a request carrying it; the pages are fetched by the operator's ordinary crawler. The string below is what a tool would match if one were ever sent.
Mozilla/5.0 (compatible; Webzio-Extended/1.0)How to block Webzio-Extended
Add this to the robots.txt at the root of your domain. Robots.txt is advisory: it works only on crawlers that choose to read it. This one is honours robots.txt according to its operator.
User-agent: Webzio-Extended
Disallow: /
To block every robots.txt control token at once, or to block training while keeping AI search citations, use the robots.txt generator.
How to verify a genuine hit
The user-agent string is self-asserted: anyone can send it from a laptop, and scrapers routinely impersonate well-known crawlers to slip past filters aimed at unknown ones.
No request carries this token; it controls how omgili's crawl is used.