Training · Anthropic
ClaudeBot
Anthropic's training crawler. Honours robots.txt and, per Anthropic, does not access content behind paywalls or CAPTCHAs. The older tokens anthropic-ai and Claude-Web are still listed in many robots.txt files and are covered here.
- Operator
- Anthropic
- Purpose
- Training crawlers
- Feeds
- Claude
- robots.txt token
ClaudeBotAlso:anthropic-aiClaude-Web- Honours robots.txt
- Honours robots.txt
- Documentation
- support.anthropic.com
What it looks like in server logs
A representative user-agent string. Version numbers change; the ClaudeBot token does not. Paste your own log line into the user-agent checker to confirm a match.
Mozilla/5.0 (compatible; ClaudeBot/1.0; [email protected])How to block ClaudeBot
Add this to the robots.txt at the root of your domain. Robots.txt is advisory: it works only on crawlers that choose to read it. This one is honours robots.txt according to its operator.
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Web
Disallow: /
To block every training crawler at once, or to block training while keeping AI search citations, use the robots.txt generator.
How to allow ClaudeBot while blocking others
A more specific User-agent group wins over a wildcard. Place this above any User-agent: * block that disallows paths, and ClaudeBot keeps full access.
User-agent: ClaudeBot
Allow: /
How to verify a genuine hit
The user-agent string is self-asserted: anyone can send it from a laptop, and scrapers routinely impersonate well-known crawlers to slip past filters aimed at unknown ones.
Anthropic publishes no machine-readable IP list, so a ClaudeBot hit cannot be verified by address. Treat the user agent as a claim, not proof.