Bot Directory
AI Assistant Fetchers
AI assistant fetchers visit a page only when a person asks an assistant such as ChatGPT, Claude or Perplexity to read it, or when the assistant decides a live page answers the question. Several operators say robots.txt may not apply to these user-requested visits.
AI Assistant Fetchers at a glance
- This group lists 30 bots.
- Blocking an AI assistant fetcher has no effect on search rankings.
- Blocking stops the assistant from reading your page when a user asks about it or pastes your link, so its answer falls back to other sources.
All AI Assistant Fetchers
| Bot | Company | Purpose | robots.txt | For GEO |
|---|---|---|---|---|
| AI2Bot-DeepResearchEval | Ai2 | AI assistantReported | Not documented | Your choice |
| amazon-QBusiness | Amazon | AI assistantReported | Not documented | Your choice |
| Amzn-User | Amazon | AI assistantOfficial docs | Not for user-requested fetches | Allow |
| ChatGPT-User | OpenAI | AI assistantOfficial docs | Not for user-requested fetches | Allow |
| Claude-User | Anthropic | AI assistantOfficial docs | Yes | Allow |
| Claude-Web | Anthropic | AI assistantReported | Not documented | Your choice |
| cohere-ai | Cohere | AI assistantReported | Not documented | Your choice |
| Diffbot-User | Diffbot | AI assistantReported | Yes | Allow |
| DoubaoBot | ByteDance | AI searchAI assistantReported | Not documented | Your choice |
| DuckAssistBot | DuckDuckGo | AI assistantOfficial docs | Yes | Allow |
| Gemini-Deep-Research | AI assistantReported | Not documented | Your choice | |
| Google-Firebase | AI assistantProduct featureReported | Not documented | Your choice | |
| Google-GeminiNotebook | AI assistantOfficial docs | Not for user-requested fetches | Allow | |
| Google-NotebookLM | AI assistantOfficial docs | Not for user-requested fetches | Allow | |
| Google-Pinpoint | AI assistantProduct featureOfficial docs | Not for user-requested fetches | Allow | |
| GoogleAgent-URLContext | AI assistantReported | Not documented | Your choice | |
| kagi-fetcher | Kagi | AI assistantReported | Not documented | Allow |
| Kimi-User | Moonshot AI | AI assistantReported | Not documented | Your choice |
| KlaviyoAIBot | Klaviyo | AI assistantReported | Yes | Your choice |
| Meta-ExternalFetcher | Meta | AI assistantOfficial docs | Not for user-requested fetches | Allow |
| MistralAI-User | Mistral AI | AI assistantOfficial docs | Yes | Allow |
| Mozilla-Tabstack | Mozilla | AI assistantProduct featureReported | Yes | Your choice |
| Perplexity-User | Perplexity | AI assistantOfficial docs | Not for user-requested fetches | Allow |
| PhindBot | Phind | AI searchAI assistantReported | Not documented | Your choice |
| QuillBot | QuillBot | AI assistantProduct featureReported | Not documented | Your choice |
| SBIntuitions-SearchBot | SB Intuitions | AI assistantOfficial docs | Yes | Allow |
| Shap-User | Parallel | AI assistantReported | Not documented | Your choice |
| TongyiBot | Alibaba | AI assistantReported | Not documented | Your choice |
| WRTNBot | Wrtn | AI assistantReported | Not documented | Your choice |
| YiyanBot | Baidu | AI assistantReported | Not documented | Your choice |
Block or allow every bot in AI Assistant Fetchers at once
The rules below name all 30 robots.txt tokens in this group. Remove the lines for any bot you want to treat differently.
robots.txt: block
# Block all AI Assistant Fetchers
User-agent: AI2Bot-DeepResearchEval
User-agent: amazon-QBusiness
User-agent: Amzn-User
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Claude-Web
User-agent: cohere-ai
User-agent: Diffbot-User
User-agent: DoubaoBot
User-agent: DuckAssistBot
User-agent: Gemini-Deep-Research
User-agent: Google-Firebase
User-agent: Google-GeminiNotebook
User-agent: Google-NotebookLM
User-agent: Google-Pinpoint
User-agent: GoogleAgent-URLContext
User-agent: kagi-fetcher
User-agent: Kimi-User
User-agent: KlaviyoAIBot
User-agent: meta-externalfetcher
User-agent: MistralAI-User
User-agent: Mozilla-Tabstack
User-agent: Perplexity-User
User-agent: PhindBot
User-agent: QuillBot
User-agent: SBIntuitions-SearchBot
User-agent: Shap-User
User-agent: TongyiBot
User-agent: WRTNBot
User-agent: YiyanBot
Disallow: /robots.txt: allow
# Allow all AI Assistant Fetchers
User-agent: AI2Bot-DeepResearchEval
User-agent: amazon-QBusiness
User-agent: Amzn-User
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Claude-Web
User-agent: cohere-ai
User-agent: Diffbot-User
User-agent: DoubaoBot
User-agent: DuckAssistBot
User-agent: Gemini-Deep-Research
User-agent: Google-Firebase
User-agent: Google-GeminiNotebook
User-agent: Google-NotebookLM
User-agent: Google-Pinpoint
User-agent: GoogleAgent-URLContext
User-agent: kagi-fetcher
User-agent: Kimi-User
User-agent: KlaviyoAIBot
User-agent: meta-externalfetcher
User-agent: MistralAI-User
User-agent: Mozilla-Tabstack
User-agent: Perplexity-User
User-agent: PhindBot
User-agent: QuillBot
User-agent: SBIntuitions-SearchBot
User-agent: Shap-User
User-agent: TongyiBot
User-agent: WRTNBot
User-agent: YiyanBot
Allow: /robots.txt: partial
# Keep all AI Assistant Fetchers out of private areas only
User-agent: AI2Bot-DeepResearchEval
User-agent: amazon-QBusiness
User-agent: Amzn-User
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Claude-Web
User-agent: cohere-ai
User-agent: Diffbot-User
User-agent: DoubaoBot
User-agent: DuckAssistBot
User-agent: Gemini-Deep-Research
User-agent: Google-Firebase
User-agent: Google-GeminiNotebook
User-agent: Google-NotebookLM
User-agent: Google-Pinpoint
User-agent: GoogleAgent-URLContext
User-agent: kagi-fetcher
User-agent: Kimi-User
User-agent: KlaviyoAIBot
User-agent: meta-externalfetcher
User-agent: MistralAI-User
User-agent: Mozilla-Tabstack
User-agent: Perplexity-User
User-agent: PhindBot
User-agent: QuillBot
User-agent: SBIntuitions-SearchBot
User-agent: Shap-User
User-agent: TongyiBot
User-agent: WRTNBot
User-agent: YiyanBot
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to all AI Assistant Fetchers
if ($http_user_agent ~* "(AI2Bot\-DeepResearchEval|amazon\-QBusiness|Amzn\-User|ChatGPT\-User|Claude\-User|Claude\-Web|cohere\-ai|Diffbot\-User|DoubaoBot|DuckAssistBot|Gemini\-Deep\-Research|Google\-Firebase|Google\-GeminiNotebook|Google\-NotebookLM|Google\-Pinpoint|GoogleAgent\-URLContext|kagi\-fetcher|Kimi\-User|KlaviyoAIBot|meta\-externalfetcher|MistralAI\-User|Mozilla\-Tabstack|Perplexity\-User|PhindBot|QuillBot|SBIntuitions\-SearchBot|Shap\-User|TongyiBot|WRTNBot|YiyanBot)") {
return 403;
}Apache
# Apache .htaccess: return 403 to all AI Assistant Fetchers
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AI2Bot\-DeepResearchEval|amazon\-QBusiness|Amzn\-User|ChatGPT\-User|Claude\-User|Claude\-Web|cohere\-ai|Diffbot\-User|DoubaoBot|DuckAssistBot|Gemini\-Deep\-Research|Google\-Firebase|Google\-GeminiNotebook|Google\-NotebookLM|Google\-Pinpoint|GoogleAgent\-URLContext|kagi\-fetcher|Kimi\-User|KlaviyoAIBot|meta\-externalfetcher|MistralAI\-User|Mozilla\-Tabstack|Perplexity\-User|PhindBot|QuillBot|SBIntuitions\-SearchBot|Shap\-User|TongyiBot|WRTNBot|YiyanBot) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "AI2Bot-DeepResearchEval") or (http.user_agent contains "amazon-QBusiness") or (http.user_agent contains "Amzn-User") or (http.user_agent contains "ChatGPT-User") or (http.user_agent contains "Claude-User") or (http.user_agent contains "Claude-Web") or (http.user_agent contains "cohere-ai") or (http.user_agent contains "Diffbot-User") or (http.user_agent contains "DoubaoBot") or (http.user_agent contains "DuckAssistBot") or (http.user_agent contains "Gemini-Deep-Research") or (http.user_agent contains "Google-Firebase") or (http.user_agent contains "Google-GeminiNotebook") or (http.user_agent contains "Google-NotebookLM") or (http.user_agent contains "Google-Pinpoint") or (http.user_agent contains "GoogleAgent-URLContext") or (http.user_agent contains "kagi-fetcher") or (http.user_agent contains "Kimi-User") or (http.user_agent contains "KlaviyoAIBot") or (http.user_agent contains "meta-externalfetcher") or (http.user_agent contains "MistralAI-User") or (http.user_agent contains "Mozilla-Tabstack") or (http.user_agent contains "Perplexity-User") or (http.user_agent contains "PhindBot") or (http.user_agent contains "QuillBot") or (http.user_agent contains "SBIntuitions-SearchBot") or (http.user_agent contains "Shap-User") or (http.user_agent contains "TongyiBot") or (http.user_agent contains "WRTNBot") or (http.user_agent contains "YiyanBot")Other bot groups
AI Training Crawlers
44 bots
Collect public pages that may be used to train AI models. Blocking them has no effect on Google or Bing rankings.
AI Search Crawlers
21 bots
Index pages so AI search products can show and cite them. Blocking them removes you from those answers.
AI Agents
14 bots
Use a browser to complete tasks for users, such as comparing, booking or buying.
Search Engine Crawlers
60 bots
Index pages for classic search engines. Blocking them removes you from search results.
Ads and Platform Fetchers
38 bots
Serve one platform feature: ad checks, feeds, verification or safety scans.
SEO Tool Crawlers
29 bots
Crawl the web for SEO platforms: backlink indexes, rank data and site audits. Blocking them hides your site from those tools.
Archives and Research Crawlers
4 bots
Save public pages for web archives, libraries and research. Blocking them stops new snapshots of your site.
Social and Messaging Previews
22 bots
Build the preview card when your link is shared in a social or messaging app. Blocking them breaks those cards.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
