Bot Directory
AI Agents
AI agents load and operate web pages in a real browser to finish a task for a user: filling a form, comparing prices, booking or buying. Their traffic often looks like a normal visitor, so many sites identify them by user agent or signed requests.
AI Agents at a glance
- This group lists 14 bots.
- Blocking an AI agent has no effect on search rankings.
- Blocking stops agents from completing tasks on your website for their users, which can cost bookings or sales that agents would have made.
All AI Agents
| Bot | Company | Purpose | robots.txt | For GEO |
|---|---|---|---|---|
| AmazonBuyForMe | Amazon | AI agentReported | Not documented | Your choice |
| ChatGPT Agent | OpenAI | AI agentReported | Yes | Allow |
| Claude-Code | Anthropic | AI agentReported | Not documented | Your choice |
| Cursor | Anysphere (Cursor) | AI agentReported | Not documented | Your choice |
| Devin | Cognition | AI agentReported | Yes | Your choice |
| FirecrawlAgent | Firecrawl | AI agentProduct featureReported | Yes | Your choice |
| Google-Agent | AI agentOfficial docs | Not for user-requested fetches | Allow | |
| Google-Gemini-CLI | AI agentReported | Not documented | Your choice | |
| GoogleAgent-Mariner | AI agentReported | Not documented | Allow | |
| Kimi-Agent | Moonshot AI | AI agentReported | Not documented | Your choice |
| Manus-User | Butterfly Effect | AI agentReported | Not documented | Your choice |
| NovaAct | Amazon | AI agentReported | Not documented | Your choice |
| opencode | SST (opencode) | AI agentReported | Not documented | Your choice |
| Trae | ByteDance | AI agentReported | Not documented | Your choice |
Block or allow every bot in AI Agents at once
The rules below name all 14 robots.txt tokens in this group. Remove the lines for any bot you want to treat differently.
robots.txt: block
# Block all AI Agents
User-agent: AmazonBuyForMe
User-agent: ChatGPT Agent
User-agent: Claude-Code
User-agent: Cursor
User-agent: Devin
User-agent: FirecrawlAgent
User-agent: Google-Agent
User-agent: Google-Gemini-CLI
User-agent: GoogleAgent-Mariner
User-agent: Kimi-Agent
User-agent: Manus-User
User-agent: NovaAct
User-agent: opencode
User-agent: Trae
Disallow: /robots.txt: allow
# Allow all AI Agents
User-agent: AmazonBuyForMe
User-agent: ChatGPT Agent
User-agent: Claude-Code
User-agent: Cursor
User-agent: Devin
User-agent: FirecrawlAgent
User-agent: Google-Agent
User-agent: Google-Gemini-CLI
User-agent: GoogleAgent-Mariner
User-agent: Kimi-Agent
User-agent: Manus-User
User-agent: NovaAct
User-agent: opencode
User-agent: Trae
Allow: /robots.txt: partial
# Keep all AI Agents out of private areas only
User-agent: AmazonBuyForMe
User-agent: ChatGPT Agent
User-agent: Claude-Code
User-agent: Cursor
User-agent: Devin
User-agent: FirecrawlAgent
User-agent: Google-Agent
User-agent: Google-Gemini-CLI
User-agent: GoogleAgent-Mariner
User-agent: Kimi-Agent
User-agent: Manus-User
User-agent: NovaAct
User-agent: opencode
User-agent: Trae
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to all AI Agents
if ($http_user_agent ~* "(AmazonBuyForMe|ChatGPT Agent|Claude\-Code|Cursor|Devin|FirecrawlAgent|Google\-Agent|Google\-Gemini\-CLI|GoogleAgent\-Mariner|Kimi\-Agent|Manus\-User|NovaAct|opencode|Trae)") {
return 403;
}Apache
# Apache .htaccess: return 403 to all AI Agents
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AmazonBuyForMe|ChatGPT Agent|Claude\-Code|Cursor|Devin|FirecrawlAgent|Google\-Agent|Google\-Gemini\-CLI|GoogleAgent\-Mariner|Kimi\-Agent|Manus\-User|NovaAct|opencode|Trae) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "AmazonBuyForMe") or (http.user_agent contains "ChatGPT Agent") or (http.user_agent contains "Claude-Code") or (http.user_agent contains "Cursor") or (http.user_agent contains "Devin") or (http.user_agent contains "FirecrawlAgent") or (http.user_agent contains "Google-Agent") or (http.user_agent contains "Google-Gemini-CLI") or (http.user_agent contains "GoogleAgent-Mariner") or (http.user_agent contains "Kimi-Agent") or (http.user_agent contains "Manus-User") or (http.user_agent contains "NovaAct") or (http.user_agent contains "opencode") or (http.user_agent contains "Trae")Other bot groups
AI Training Crawlers
44 bots
Collect public pages that may be used to train AI models. Blocking them has no effect on Google or Bing rankings.
AI Search Crawlers
21 bots
Index pages so AI search products can show and cite them. Blocking them removes you from those answers.
AI Assistant Fetchers
30 bots
Open a page when a user asks an AI assistant about it. They act for one person, not for an index.
Search Engine Crawlers
60 bots
Index pages for classic search engines. Blocking them removes you from search results.
Ads and Platform Fetchers
38 bots
Serve one platform feature: ad checks, feeds, verification or safety scans.
SEO Tool Crawlers
29 bots
Crawl the web for SEO platforms: backlink indexes, rank data and site audits. Blocking them hides your site from those tools.
Archives and Research Crawlers
4 bots
Save public pages for web archives, libraries and research. Blocking them stops new snapshots of your site.
Social and Messaging Previews
22 bots
Build the preview card when your link is shared in a social or messaging app. Blocking them breaks those cards.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
