AI crawler directory
Every AI crawler, by name
49 AI crawlers, fetchers and agents: who runs each one, what it feeds, the exact User-agent token for robots.txt, and whether a hit can be verified. Updated .
Grouped by what the crawl is for, because that is the decision a site owner actually makes: block training, keep citations, or both.
Training crawlers
Build the corpus a model is trained on. Blocking one keeps future models from learning from your pages; it has no effect on whether an AI product can cite you today.
| Crawler | Operator | Feeds | robots.txt token | Robots.txt |
|---|---|---|---|---|
| GPTBot | OpenAI | ChatGPT | GPTBot | Honours robots.txt |
| ClaudeBot | Anthropic | Claude | ClaudeBot | Honours robots.txt |
| GoogleOther | Google (other products) | GoogleOther | Honours robots.txt | |
| Amazonbot | Amazon | Alexa | Amazonbot | Honours robots.txt |
| Meta-ExternalAgent | Meta | Meta AI | Meta-ExternalAgent | Honours robots.txt |
| CCBot | Common Crawl | Common Crawl corpus | CCBot | Honours robots.txt |
| Bytespider | ByteDance | Doubao / TikTok | Bytespider | Undocumented |
| Cohere | Cohere | Cohere | cohere-ai | Undocumented |
| Diffbot | Diffbot | Diffbot KG | Diffbot | Undocumented |
| PanguBot | Huawei | Huawei PanGu | PanguBot | Undocumented |
| ImagesiftBot | The Hive | ImageSift | ImagesiftBot | Honours robots.txt |
| AI2Bot | Allen Institute for AI | Allen Institute | AI2Bot | Honours robots.txt |
| Omgili | Webz.io | Webz.io corpus | omgili | Honours robots.txt |
| YandexAdditional | Yandex | Yandex AI | YandexAdditional | Honours robots.txt |
| SBIntuitionsBot | SB Intuitions (SoftBank) | SB Intuitions | SBIntuitionsBot | Honours robots.txt |
| ICC-Crawler | NICT (Japan) | NICT corpus | ICC-Crawler | Honours robots.txt |
| VelenPublicWebCrawler | Velen | Velen corpus | VelenPublicWebCrawler | Undocumented |
| img2dataset | Open-source tool | Dataset builder | img2dataset | May ignore robots.txt |
| aiHitBot | aiHit | aiHit corpus | aiHitBot | Undocumented |
| Crawlspace | Crawlspace | Crawlspace corpus | Crawlspace | Undocumented |
| Poseidon Research | Poseidon Research | Poseidon corpus | Poseidon Research Crawler | Undocumented |
AI search indexers
Build the index an AI answer engine retrieves from. Blocking one removes your site from that product's citations — the traffic source most sites want to keep.
| Crawler | Operator | Feeds | robots.txt token | Robots.txt |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT Search | OAI-SearchBot | Honours robots.txt |
| Claude-SearchBot | Anthropic | Claude Search | Claude-SearchBot | Honours robots.txt |
| PerplexityBot | Perplexity | Perplexity | PerplexityBot | Honours robots.txt |
| Google-CloudVertexBot | Vertex AI grounding | Google-CloudVertexBot | Honours robots.txt | |
| DuckAssistBot | DuckDuckGo | DuckDuckGo AI | DuckAssistBot | Honours robots.txt |
| PetalBot | Huawei | Petal Search | PetalBot | Honours robots.txt |
| YouBot | You.com | You.com | YouBot | Honours robots.txt |
| Timpibot | Timpi | Timpi | Timpibot | Undocumented |
| xAI-Bot | xAI | Grok | xAI-Bot | Undocumented |
| PhindBot | Phind | Phind | PhindBot | Undocumented |
| Kagibot | Kagi | Kagi Assistant | Kagibot | Honours robots.txt |
| Bravebot | Brave | Brave Leo | Bravebot | Honours robots.txt |
| iAskSpider | iAsk.ai | iAsk.ai | iAskSpider | Undocumented |
| Andibot | Andi | Andi | Andibot | Undocumented |
User-triggered fetchers
Fetch a page because a person asked the assistant about it or pasted the link. Several operators state these fetchers act on the user's behalf and do not check robots.txt.
| Crawler | Operator | Feeds | robots.txt token | Robots.txt |
|---|---|---|---|---|
| ChatGPT-User | OpenAI | ChatGPT (live browse) | ChatGPT-User | Honours robots.txt |
| Claude-User | Anthropic | Claude (live browse) | Claude-User | Honours robots.txt |
| Perplexity-User | Perplexity | Perplexity (live browse) | Perplexity-User | May ignore robots.txt |
| Meta-ExternalFetcher | Meta | Meta AI (link fetch) | Meta-ExternalFetcher | May ignore robots.txt |
| MistralAI-User | Mistral AI | Le Chat | MistralAI-User | Honours robots.txt |
| SemrushBot-OCOB | Semrush | Semrush AI corpus | SemrushBot-OCOB | Honours robots.txt |
Agentic browsers
Browse and act on one person's instruction — comparing prices, filling forms, completing a task. Closer to a visit than a crawl, but they do not run analytics scripts.
| Crawler | Operator | Feeds | robots.txt token | Robots.txt |
|---|---|---|---|---|
| ChatGPT-Agent | OpenAI | ChatGPT (agent) | ChatGPT-Agent | Honours robots.txt |
| GoogleAgent-Mariner | Project Mariner (agent) | GoogleAgent-Mariner | Undocumented | |
| NovaAct | Amazon | Amazon Nova Act (agent) | NovaAct | Undocumented |
| Devin | Cognition | Devin (agent) | Devin | Undocumented |
| FirecrawlAgent | Firecrawl | Firecrawl (agent scraping) | FirecrawlAgent | Undocumented |
Robots.txt control tokens
Not a crawler you will see in logs. A token the operator reads in robots.txt to decide whether pages its ordinary crawler already fetched may be used for AI features.
| Crawler | Operator | Feeds | robots.txt token | Robots.txt |
|---|---|---|---|---|
| Google-Extended | Gemini / AI Overviews | Google-Extended | Honours robots.txt | |
| Applebot-Extended | Apple | Apple Intelligence | Applebot-Extended | Honours robots.txt |
| Webzio-Extended | Webz.io | Webz.io corpus | Webzio-Extended | Honours robots.txt |
How this list is maintained
Every entry corresponds to a signature in the classifier that runs on NumberHill's ingest endpoint, so the directory and the product cannot disagree about what a user agent is. A test fails the build if either side gains an entry the other lacks.
Robots.txt behaviour is recorded as the operator documents it, with a link where one exists. “Undocumented” means the operator has published nothing either way, not that the crawler misbehaves. Sample user-agent strings are representative of what appears in server logs; version numbers change.
To report a missing crawler or a correction, email [email protected] with the user-agent string and the operator's documentation page.
The whole list is also available as ai-crawlers.json with open CORS, and the generator's presets as a plain ai-robots.txt you can curl into a build step.
Common questions
What is the difference between an AI training crawler and an AI search crawler?
A training crawler (GPTBot, ClaudeBot, CCBot) collects pages to train future models; blocking it has no effect on whether a product cites you today. An AI search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot) builds the index an answer engine retrieves from; blocking it removes your site from that product's citations and the referral traffic they send.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended is a robots.txt control token, not a crawler. Disallowing it tells Google not to use pages Googlebot already fetched for Gemini training and grounding. Google Search indexing and AI Overviews follow the ordinary Googlebot rules and are unaffected.
Which AI crawlers ignore robots.txt?
By their operators' own documentation, user-triggered fetchers such as Perplexity-User and Meta-ExternalFetcher may bypass robots.txt because a person requested the specific page. Bytespider is widely reported by site operators to continue crawling after being disallowed. Where an operator publishes nothing, this directory says so rather than guessing.
How can I tell whether a hit claiming to be GPTBot is genuine?
OpenAI, Perplexity, Google, DuckDuckGo, Mistral and Kagi publish the IP ranges their crawlers use; a request from outside those ranges is spoofed. Amazon, Apple, Yandex and Huawei document reverse-DNS verification. Anthropic publishes no machine-readable list, so a ClaudeBot hit cannot be verified by address.
Should I block AI crawlers at all?
It depends on what the site is for. A publisher whose revenue is page views loses when models answer from its content without a click. A business that sells something usually benefits from being cited: an engine that cannot read a page cannot recommend it. Most commercial sites do best blocking training crawlers and allowing AI search and user-triggered fetchers.