Skip to content

robots.txt generator

Block AI crawlers without losing AI citations

Pick the crawlers to block from all 49 in the directory, grouped by what the crawl is for. The default blocks training crawlers and leaves AI search indexers alone, so future models cannot learn from your pages but ChatGPT, Perplexity and Claude can still cite them.

Training crawlers

Training crawlers

Build the corpus a model is trained on. Blocking one keeps future models from learning from your pages; it has no effect on whether an AI product can cite you today.

AI search indexers

AI search indexers

Build the index an AI answer engine retrieves from. Blocking one removes your site from that product's citations — the traffic source most sites want to keep.

User-triggered fetchers

User-triggered fetchers

Fetch a page because a person asked the assistant about it or pasted the link. Several operators state these fetchers act on the user's behalf and do not check robots.txt.

Agentic browsers

Agentic browsers

Browse and act on one person's instruction — comparing prices, filling forms, completing a task. Closer to a visit than a crawl, but they do not run analytics scripts.

Robots.txt control tokens

Robots.txt control tokens

Not a crawler you will see in logs. A token the operator reads in robots.txt to decide whether pages its ordinary crawler already fetched may be used for AI features.

robots.txt · 24 crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Applebot-Extended
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: FacebookBot
User-agent: CCBot
User-agent: Bytespider
User-agent: TikTokSpider
User-agent: cohere-ai
User-agent: Diffbot
User-agent: PanguBot
User-agent: ImagesiftBot
User-agent: AI2Bot
User-agent: AI2Bot-Dolma
User-agent: omgili
User-agent: omgilibot
User-agent: Webzio-Extended
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
User-agent: SBIntuitionsBot
User-agent: ICC-Crawler
User-agent: VelenPublicWebCrawler
User-agent: img2dataset
User-agent: aiHitBot
User-agent: Crawlspace
User-agent: Poseidon Research Crawler
Disallow: /

Each selected crawler has its own page: GPTBot, ClaudeBot, Google-Extended, GoogleOther, Applebot-Extended and 19 more.

The rules most commercial sites want

Training crawlers and the control tokens, nothing else. AI search indexers and on-demand fetchers stay allowed, so an assistant that is asked about your product can still read the page and link to it.

robots.txt
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: GoogleOther
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: Diffbot
User-agent: PanguBot
User-agent: ImagesiftBot
User-agent: AI2Bot
User-agent: omgili
User-agent: YandexAdditional
User-agent: SBIntuitionsBot
User-agent: ICC-Crawler
User-agent: VelenPublicWebCrawler
User-agent: img2dataset
User-agent: aiHitBot
User-agent: Crawlspace
User-agent: Poseidon Research Crawler
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Webzio-Extended
Disallow: /

Place these groups above any User-agent: * block. A crawler obeys the most specific group that names it, and only that group, so a wildcard Disallow further down does not change what the named agents may fetch.

Common questions

Will blocking AI crawlers in robots.txt hurt my Google rankings?

No, as long as you do not block Googlebot. Google-Extended is a separate token that only controls AI use of already-crawled pages. Every rule this generator emits names a specific AI agent; none touches a search engine's main crawler.

Is robots.txt enough to stop AI crawlers?

It stops the ones that honour it, which includes every major operator: OpenAI, Anthropic, Google, Apple, Amazon, Common Crawl and Meta's training crawler. User-triggered fetchers and a few high-volume crawlers document or are reported to ignore it. For those, match the user agent at your CDN or server and return 403, or use a bot-management product.

Where does the file go?

At the root of the host: https://example.com/robots.txt. Every subdomain needs its own. Changes are picked up the next time a crawler fetches the file, which for large operators is usually within a day.

Should I block on-demand fetchers like ChatGPT-User?

Usually not. They fetch a page because a person asked about it, and the result often includes a link back to you. Blocking them makes an assistant unable to read your page when a user pastes the URL, and several ignore robots.txt anyway.

robots.txt Generator for AI Crawlers — NumberHill