Bot Directory
AI Training Crawlers
AI training crawlers download public web pages so an AI company can build or improve its language models. They do not decide whether you rank in search, and most of them do not send visitors back. Blocking one is a content-protection choice, not an SEO risk.
AI Training Crawlers at a glance
- This group lists 44 bots.
- Blocking an AI training crawler has no effect on Google or Bing rankings.
- Blocking keeps your future pages out of that company's training data, so its models may know less about your brand over time. It does not remove you from that company's AI search, which uses a separate bot.
All AI Training Crawlers
| Bot | Company | Purpose | robots.txt | For GEO |
|---|---|---|---|---|
| AI2Bot | Ai2 | AI trainingOfficial docs | Yes | Your choice |
| Ai2Bot-Dolma | Ai2 | AI trainingReported | Yes | Your choice |
| Amazonbot | Amazon | AI trainingSearch indexOfficial docs | Yes | Your choice |
| anthropic-ai | Anthropic | AI trainingReported | Not documented | Your choice |
| ApifyWebsiteContentCrawler | Apify | Product featureReported | Not documented | Your choice |
| Applebot-Extended | Apple | AI trainingOfficial docs | Yes | Your choice |
| Brightbot | Bright Data | AI trainingProduct featureOfficial docs | Yes | Your choice |
| Bytespider | ByteDance | AI trainingReported | No | Consider blocking |
| CCBot | Common Crawl | AI trainingOfficial docs | Yes | Your choice |
| ChatGLM-Spider | Zhipu AI | AI trainingReported | Not documented | Your choice |
| ClaudeBot | Anthropic | AI trainingOfficial docs | Yes | Your choice |
| CloudflareBrowserRenderingCrawler | Cloudflare | Product featureOfficial docs | Yes | Your choice |
| cohere-training-data-crawler | Cohere | AI trainingReported | Not documented | Your choice |
| Cotoyogi | ROIS | AI trainingOfficial docs | Yes | Your choice |
| DeepSeekBot | DeepSeek | AI trainingReported | No | Consider blocking |
| Diffbot | Diffbot | AI trainingProduct featureReported | Not documented | Your choice |
| ERNIEBot | Baidu | AI trainingReported | Not documented | Your choice |
| FacebookBot | Meta | AI trainingReported | Yes | Your choice |
| Factset_spyderbot | FactSet | AI trainingReported | Not documented | Your choice |
| Google-CloudVertexBot | Product featureOfficial docs | Yes | Your choice | |
| Google-Extended | AI trainingOfficial docs | Yes | Your choice | |
| GoogleOther | AI trainingProduct featureOfficial docs | Yes | Your choice | |
| GoogleOther-Image | AI trainingProduct featureOfficial docs | Yes | Your choice | |
| GoogleOther-Video | AI trainingProduct featureOfficial docs | Yes | Your choice | |
| GPTBot | OpenAI | AI trainingOfficial docs | Yes | Your choice |
| Hunyuan | Tencent | AI trainingReported | Not documented | Your choice |
| ICC-Crawler | NICT | AI trainingReported | Yes | Your choice |
| ImagesiftBot | ImageSift | AI trainingProduct featureOfficial docs | Yes | Your choice |
| KimiBot | Moonshot AI | AI trainingOfficial docs | Yes | Your choice |
| Meta-ExternalAgent | Meta | AI trainingOfficial docs | Yes | Your choice |
| MistralAI-Training | Mistral AI | AI trainingOfficial docs | Yes | Your choice |
| MoonshotBot | Moonshot AI | AI trainingReported | Not documented | Your choice |
| omgili | Webz.io | AI trainingReported | Yes | Your choice |
| omgilibot | Webz.io | AI trainingOfficial docs | Yes | Your choice |
| PanguBot | Huawei | AI trainingReported | Not documented | Your choice |
| QwenBot | Alibaba | AI trainingReported | Not documented | Your choice |
| Reflectionbot | Reflection AI | AI trainingReported | Not documented | Your choice |
| SBIntuitionsBot | SB Intuitions | AI trainingOfficial docs | Yes | Your choice |
| TikTokSpider | ByteDance | AI trainingReported | Not documented | Your choice |
| Timpibot | Timpi | AI trainingReported | Not documented | Your choice |
| Webzio-Extended | Webz.io | AI trainingReported | Not documented | Your choice |
| xAI-Grok | xAI | AI trainingReported | Not documented | Your choice |
| YandexAdditional | Yandex | AI trainingAI searchOfficial docs | Yes | Your choice |
| YiBot | 01.AI | AI trainingReported | Not documented | Your choice |
Block or allow every bot in AI Training Crawlers at once
The rules below name all 44 robots.txt tokens in this group. Remove the lines for any bot you want to treat differently.
robots.txt: block
# Block all AI Training Crawlers
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: ApifyWebsiteContentCrawler
User-agent: Applebot-Extended
User-agent: Brightbot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGLM-Spider
User-agent: ClaudeBot
User-agent: CloudflareBrowserRenderingCrawler
User-agent: cohere-training-data-crawler
User-agent: Cotoyogi
User-agent: DeepSeekBot
User-agent: Diffbot
User-agent: ERNIEBot
User-agent: FacebookBot
User-agent: Factset_spyderbot
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: Hunyuan
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: KimiBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: MoonshotBot
User-agent: omgili
User-agent: omgilibot
User-agent: PanguBot
User-agent: QwenBot
User-agent: Reflectionbot
User-agent: SBIntuitionsBot
User-agent: TikTokSpider
User-agent: Timpibot
User-agent: Webzio-Extended
User-agent: xAI-Grok
User-agent: YandexAdditional
User-agent: YiBot
Disallow: /robots.txt: allow
# Allow all AI Training Crawlers
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: ApifyWebsiteContentCrawler
User-agent: Applebot-Extended
User-agent: Brightbot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGLM-Spider
User-agent: ClaudeBot
User-agent: CloudflareBrowserRenderingCrawler
User-agent: cohere-training-data-crawler
User-agent: Cotoyogi
User-agent: DeepSeekBot
User-agent: Diffbot
User-agent: ERNIEBot
User-agent: FacebookBot
User-agent: Factset_spyderbot
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: Hunyuan
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: KimiBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: MoonshotBot
User-agent: omgili
User-agent: omgilibot
User-agent: PanguBot
User-agent: QwenBot
User-agent: Reflectionbot
User-agent: SBIntuitionsBot
User-agent: TikTokSpider
User-agent: Timpibot
User-agent: Webzio-Extended
User-agent: xAI-Grok
User-agent: YandexAdditional
User-agent: YiBot
Allow: /robots.txt: partial
# Keep all AI Training Crawlers out of private areas only
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Amazonbot
User-agent: anthropic-ai
User-agent: ApifyWebsiteContentCrawler
User-agent: Applebot-Extended
User-agent: Brightbot
User-agent: Bytespider
User-agent: CCBot
User-agent: ChatGLM-Spider
User-agent: ClaudeBot
User-agent: CloudflareBrowserRenderingCrawler
User-agent: cohere-training-data-crawler
User-agent: Cotoyogi
User-agent: DeepSeekBot
User-agent: Diffbot
User-agent: ERNIEBot
User-agent: FacebookBot
User-agent: Factset_spyderbot
User-agent: Google-CloudVertexBot
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: Hunyuan
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: KimiBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: MoonshotBot
User-agent: omgili
User-agent: omgilibot
User-agent: PanguBot
User-agent: QwenBot
User-agent: Reflectionbot
User-agent: SBIntuitionsBot
User-agent: TikTokSpider
User-agent: Timpibot
User-agent: Webzio-Extended
User-agent: xAI-Grok
User-agent: YandexAdditional
User-agent: YiBot
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to all AI Training Crawlers
if ($http_user_agent ~* "(AI2Bot|Ai2Bot\-Dolma|Amazonbot|anthropic\-ai|ApifyWebsiteContentCrawler|Applebot\-Extended|Brightbot|Bytespider|CCBot|ChatGLM\-Spider|ClaudeBot|CloudflareBrowserRenderingCrawler|cohere\-training\-data\-crawler|Cotoyogi|DeepSeekBot|Diffbot|ERNIEBot|FacebookBot|Factset_spyderbot|Google\-CloudVertexBot|Google\-Extended|GoogleOther|GoogleOther\-Image|GoogleOther\-Video|GPTBot|Hunyuan|ICC\-Crawler|ImagesiftBot|KimiBot|meta\-externalagent|MistralAI\-Training|MoonshotBot|omgili|omgilibot|PanguBot|QwenBot|Reflectionbot|SBIntuitionsBot|TikTokSpider|Timpibot|Webzio\-Extended|xAI\-Grok|YandexAdditional|YiBot)") {
return 403;
}Apache
# Apache .htaccess: return 403 to all AI Training Crawlers
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AI2Bot|Ai2Bot\-Dolma|Amazonbot|anthropic\-ai|ApifyWebsiteContentCrawler|Applebot\-Extended|Brightbot|Bytespider|CCBot|ChatGLM\-Spider|ClaudeBot|CloudflareBrowserRenderingCrawler|cohere\-training\-data\-crawler|Cotoyogi|DeepSeekBot|Diffbot|ERNIEBot|FacebookBot|Factset_spyderbot|Google\-CloudVertexBot|Google\-Extended|GoogleOther|GoogleOther\-Image|GoogleOther\-Video|GPTBot|Hunyuan|ICC\-Crawler|ImagesiftBot|KimiBot|meta\-externalagent|MistralAI\-Training|MoonshotBot|omgili|omgilibot|PanguBot|QwenBot|Reflectionbot|SBIntuitionsBot|TikTokSpider|Timpibot|Webzio\-Extended|xAI\-Grok|YandexAdditional|YiBot) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "AI2Bot") or (http.user_agent contains "Ai2Bot-Dolma") or (http.user_agent contains "Amazonbot") or (http.user_agent contains "anthropic-ai") or (http.user_agent contains "ApifyWebsiteContentCrawler") or (http.user_agent contains "Applebot-Extended") or (http.user_agent contains "Brightbot") or (http.user_agent contains "Bytespider") or (http.user_agent contains "CCBot") or (http.user_agent contains "ChatGLM-Spider") or (http.user_agent contains "ClaudeBot") or (http.user_agent contains "CloudflareBrowserRenderingCrawler") or (http.user_agent contains "cohere-training-data-crawler") or (http.user_agent contains "Cotoyogi") or (http.user_agent contains "DeepSeekBot") or (http.user_agent contains "Diffbot") or (http.user_agent contains "ERNIEBot") or (http.user_agent contains "FacebookBot") or (http.user_agent contains "Factset_spyderbot") or (http.user_agent contains "Google-CloudVertexBot") or (http.user_agent contains "Google-Extended") or (http.user_agent contains "GoogleOther") or (http.user_agent contains "GoogleOther-Image") or (http.user_agent contains "GoogleOther-Video") or (http.user_agent contains "GPTBot") or (http.user_agent contains "Hunyuan") or (http.user_agent contains "ICC-Crawler") or (http.user_agent contains "ImagesiftBot") or (http.user_agent contains "KimiBot") or (http.user_agent contains "meta-externalagent") or (http.user_agent contains "MistralAI-Training") or (http.user_agent contains "MoonshotBot") or (http.user_agent contains "omgili") or (http.user_agent contains "omgilibot") or (http.user_agent contains "PanguBot") or (http.user_agent contains "QwenBot") or (http.user_agent contains "Reflectionbot") or (http.user_agent contains "SBIntuitionsBot") or (http.user_agent contains "TikTokSpider") or (http.user_agent contains "Timpibot") or (http.user_agent contains "Webzio-Extended") or (http.user_agent contains "xAI-Grok") or (http.user_agent contains "YandexAdditional") or (http.user_agent contains "YiBot")Other bot groups
AI Search Crawlers
21 bots
Index pages so AI search products can show and cite them. Blocking them removes you from those answers.
AI Assistant Fetchers
30 bots
Open a page when a user asks an AI assistant about it. They act for one person, not for an index.
AI Agents
14 bots
Use a browser to complete tasks for users, such as comparing, booking or buying.
Search Engine Crawlers
60 bots
Index pages for classic search engines. Blocking them removes you from search results.
Ads and Platform Fetchers
38 bots
Serve one platform feature: ad checks, feeds, verification or safety scans.
SEO Tool Crawlers
29 bots
Crawl the web for SEO platforms: backlink indexes, rank data and site audits. Blocking them hides your site from those tools.
Archives and Research Crawlers
4 bots
Save public pages for web archives, libraries and research. Blocking them stops new snapshots of your site.
Social and Messaging Previews
22 bots
Build the preview card when your link is shared in a social or messaging app. Blocking them breaks those cards.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
