Training · Common Crawl
CCBot
Builds the Common Crawl open dataset, which is the raw material for a large share of published language-model training sets. Blocking it removes future crawls from the corpus but not the archives already published.
- Operator
- Common Crawl
- Purpose
- Training crawlers
- Feeds
- Common Crawl corpus
- robots.txt token
CCBot- Honours robots.txt
- Honours robots.txt
- Documentation
- commoncrawl.org
What it looks like in server logs
A representative user-agent string. Version numbers change; the CCBot token does not. Paste your own log line into the user-agent checker to confirm a match.
CCBot/2.0 (https://commoncrawl.org/faq/)How to block CCBot
Add this to the robots.txt at the root of your domain. Robots.txt is advisory: it works only on crawlers that choose to read it. This one is honours robots.txt according to its operator.
User-agent: CCBot
Disallow: /
To block every training crawler at once, or to block training while keeping AI search citations, use the robots.txt generator.
How to allow CCBot while blocking others
A more specific User-agent group wins over a wildcard. Place this above any User-agent: * block that disallows paths, and CCBot keeps full access.
User-agent: CCBot
Allow: /
How to verify a genuine hit
The user-agent string is self-asserted: anyone can send it from a laptop, and scrapers routinely impersonate well-known crawlers to slip past filters aimed at unknown ones.
Common Crawl documents that it crawls from AWS; reverse DNS points at amazonaws.com hosts.