Skip to content

Control token · Webz.io

Webzio-Extended

A robots.txt token that lets a site stay in Webz.io's general dataset while opting out of the subset sold for AI training.

Operator
Webz.io
Feeds
Webz.io corpus
robots.txt token
Webzio-Extended
Honours robots.txt
Honours robots.txt
Documentation
webz.io

Does it appear in server logs?

No. Webzio-Extended is a token the operator reads in robots.txt, not a user agent that fetches pages. You will never see a request carrying it; the pages are fetched by the operator's ordinary crawler. The string below is what a tool would match if one were ever sent.

user-agent
Mozilla/5.0 (compatible; Webzio-Extended/1.0)

How to block Webzio-Extended

Add this to the robots.txt at the root of your domain. Robots.txt is advisory: it works only on crawlers that choose to read it. This one is honours robots.txt according to its operator.

robots.txt
User-agent: Webzio-Extended
Disallow: /

To block every robots.txt control token at once, or to block training while keeping AI search citations, use the robots.txt generator.

How to verify a genuine hit

The user-agent string is self-asserted: anyone can send it from a laptop, and scrapers routinely impersonate well-known crawlers to slip past filters aimed at unknown ones.

No request carries this token; it controls how omgili's crawl is used.

Also run by Webz.io

Other robots.txt control tokens

Full directory of 49 AI crawlers