Bot Directory
AI Search Crawlers
AI search crawlers build the index behind AI answer engines such as ChatGPT search, Claude search and Perplexity. A page these bots cannot reach cannot be cited or linked in those answers. For Generative Engine Optimization (GEO), these are the bots to allow first.
AI Search Crawlers at a glance
- This group lists 21 bots.
- Blocking an AI search crawler has no effect on Google or Bing rankings.
- Blocking removes your pages from that AI search index, so its answers can no longer cite or link to your website.
All AI Search Crawlers
| Bot | Company | Purpose | robots.txt | For GEO |
|---|---|---|---|---|
| amazon-kendra | Amazon | AI searchProduct featureOfficial docs | Yes | Your choice |
| Amzn-SearchBot | Amazon | AI searchOfficial docs | Yes | Allow |
| Andibot | Andi | AI searchReported | Not documented | Your choice |
| atlassian-bot | Atlassian | AI searchAI assistantReported | Yes | Your choice |
| AzureAI-SearchBot | Microsoft | AI searchReported | Not documented | Your choice |
| bedrockbot | Amazon | AI searchProduct featureOfficial docs | Yes | Your choice |
| Claude-SearchBot | Anthropic | AI searchOfficial docs | Yes | Allow |
| Cloudflare-AutoRAG | Cloudflare | AI searchProduct featureReported | Yes | Your choice |
| ExaSearchBot | Exa | AI searchReported | Not documented | Allow |
| GrokBot | xAI | AI trainingAI searchReported | Not documented | Your choice |
| iaskspider | iAsk | AI searchReported | No | Consider blocking |
| Kimi-SearchBot | Moonshot AI | AI searchOfficial docs | Yes | Allow |
| LinerBot | Liner | AI searchAI assistantReported | Not documented | Allow |
| LinkupBot | Linkup | AI searchReported | Not documented | Allow |
| Meta-WebIndexer | Meta | AI searchOfficial docs | Yes | Allow |
| MistralAI-Index | Mistral AI | AI searchOfficial docs | Yes | Allow |
| OAI-SearchBot | OpenAI | AI searchOfficial docs | Yes | Allow |
| PerplexityBot | Perplexity | AI searchOfficial docs | Yes | Allow |
| ShapBot | Parallel | AI searchOfficial docs | Yes | Allow |
| TavilyBot | Tavily | AI searchReported | Not documented | Allow |
| YouBot | You.com | AI searchReported | Yes | Allow |
Block or allow every bot in AI Search Crawlers at once
The rules below name all 21 robots.txt tokens in this group. Remove the lines for any bot you want to treat differently.
robots.txt: block
# Block all AI Search Crawlers
User-agent: amazon-kendra
User-agent: Amzn-SearchBot
User-agent: Andibot
User-agent: atlassian-bot
User-agent: AzureAI-SearchBot
User-agent: bedrockbot
User-agent: Claude-SearchBot
User-agent: Cloudflare-AutoRAG
User-agent: ExaSearchBot
User-agent: GrokBot
User-agent: iaskspider
User-agent: Kimi-SearchBot
User-agent: LinerBot
User-agent: LinkupBot
User-agent: meta-webindexer
User-agent: MistralAI-Index
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: ShapBot
User-agent: TavilyBot
User-agent: YouBot
Disallow: /robots.txt: allow
# Allow all AI Search Crawlers
User-agent: amazon-kendra
User-agent: Amzn-SearchBot
User-agent: Andibot
User-agent: atlassian-bot
User-agent: AzureAI-SearchBot
User-agent: bedrockbot
User-agent: Claude-SearchBot
User-agent: Cloudflare-AutoRAG
User-agent: ExaSearchBot
User-agent: GrokBot
User-agent: iaskspider
User-agent: Kimi-SearchBot
User-agent: LinerBot
User-agent: LinkupBot
User-agent: meta-webindexer
User-agent: MistralAI-Index
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: ShapBot
User-agent: TavilyBot
User-agent: YouBot
Allow: /robots.txt: partial
# Keep all AI Search Crawlers out of private areas only
User-agent: amazon-kendra
User-agent: Amzn-SearchBot
User-agent: Andibot
User-agent: atlassian-bot
User-agent: AzureAI-SearchBot
User-agent: bedrockbot
User-agent: Claude-SearchBot
User-agent: Cloudflare-AutoRAG
User-agent: ExaSearchBot
User-agent: GrokBot
User-agent: iaskspider
User-agent: Kimi-SearchBot
User-agent: LinerBot
User-agent: LinkupBot
User-agent: meta-webindexer
User-agent: MistralAI-Index
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: ShapBot
User-agent: TavilyBot
User-agent: YouBot
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to all AI Search Crawlers
if ($http_user_agent ~* "(amazon\-kendra|Amzn\-SearchBot|Andibot|atlassian\-bot|AzureAI\-SearchBot|bedrockbot|Claude\-SearchBot|Cloudflare\-AutoRAG|ExaSearchBot|GrokBot|iaskspider|Kimi\-SearchBot|LinerBot|LinkupBot|meta\-webindexer|MistralAI\-Index|OAI\-SearchBot|PerplexityBot|ShapBot|TavilyBot|YouBot)") {
return 403;
}Apache
# Apache .htaccess: return 403 to all AI Search Crawlers
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (amazon\-kendra|Amzn\-SearchBot|Andibot|atlassian\-bot|AzureAI\-SearchBot|bedrockbot|Claude\-SearchBot|Cloudflare\-AutoRAG|ExaSearchBot|GrokBot|iaskspider|Kimi\-SearchBot|LinerBot|LinkupBot|meta\-webindexer|MistralAI\-Index|OAI\-SearchBot|PerplexityBot|ShapBot|TavilyBot|YouBot) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "amazon-kendra") or (http.user_agent contains "Amzn-SearchBot") or (http.user_agent contains "Andibot") or (http.user_agent contains "atlassian-bot") or (http.user_agent contains "AzureAI-SearchBot") or (http.user_agent contains "bedrockbot") or (http.user_agent contains "Claude-SearchBot") or (http.user_agent contains "Cloudflare-AutoRAG") or (http.user_agent contains "ExaSearchBot") or (http.user_agent contains "GrokBot") or (http.user_agent contains "iaskspider") or (http.user_agent contains "Kimi-SearchBot") or (http.user_agent contains "LinerBot") or (http.user_agent contains "LinkupBot") or (http.user_agent contains "meta-webindexer") or (http.user_agent contains "MistralAI-Index") or (http.user_agent contains "OAI-SearchBot") or (http.user_agent contains "PerplexityBot") or (http.user_agent contains "ShapBot") or (http.user_agent contains "TavilyBot") or (http.user_agent contains "YouBot")Other bot groups
AI Training Crawlers
44 bots
Collect public pages that may be used to train AI models. Blocking them has no effect on Google or Bing rankings.
AI Assistant Fetchers
30 bots
Open a page when a user asks an AI assistant about it. They act for one person, not for an index.
AI Agents
14 bots
Use a browser to complete tasks for users, such as comparing, booking or buying.
Search Engine Crawlers
60 bots
Index pages for classic search engines. Blocking them removes you from search results.
Ads and Platform Fetchers
38 bots
Serve one platform feature: ad checks, feeds, verification or safety scans.
SEO Tool Crawlers
29 bots
Crawl the web for SEO platforms: backlink indexes, rank data and site audits. Blocking them hides your site from those tools.
Archives and Research Crawlers
4 bots
Save public pages for web archives, libraries and research. Blocking them stops new snapshots of your site.
Social and Messaging Previews
22 bots
Build the preview card when your link is shared in a social or messaging app. Blocking them breaks those cards.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
