Bots by company
Google bots and crawlers
Google runs Google Search, Gemini and Google Ads. It documents common crawlers, special-case crawlers and user-triggered fetchers, plus the Google-Extended control token for Gemini training.
Google bots at a glance
- Google runs 32 bots listed in this directory.
- They belong to: Ads and Platform Fetchers, SEO Tool Crawlers, AI Assistant Fetchers, AI Agents, AI Training Crawlers, Search Engine Crawlers, Social and Messaging Previews.
- Each Google bot has its own robots.txt token, so each purpose can be allowed or blocked separately.
- Website: www.google.com
Every Google bot
| Bot | Group | Purpose | robots.txt | For GEO |
|---|---|---|---|---|
| AdsBot-Google | Ads and Platform Fetchers | AdsOfficial docs | Yes | Allow |
| AdsBot-Google-Mobile | Ads and Platform Fetchers | AdsOfficial docs | Yes | Allow |
| APIs-Google | Ads and Platform Fetchers | Product featureOfficial docs | Yes | Allow |
| Chrome-Lighthouse | SEO Tool Crawlers | Product featureReported | Not documented | Allow |
| FeedFetcher-Google | Ads and Platform Fetchers | Product featureOfficial docs | Not for user-requested fetches | Allow |
| Gemini-Deep-Research | AI Assistant Fetchers | AI assistantReported | Not documented | Your choice |
| Google-Agent | AI Agents | AI agentOfficial docs | Not for user-requested fetches | Allow |
| Google-CloudVertexBot | AI Training Crawlers | Product featureOfficial docs | Yes | Your choice |
| Google-CWS | Ads and Platform Fetchers | Product featureOfficial docs | Not for user-requested fetches | Allow |
| Google-Extended | AI Training Crawlers | AI trainingOfficial docs | Yes | Your choice |
| Google-Firebase | AI Assistant Fetchers | AI assistantProduct featureReported | Not documented | Your choice |
| Google-Gemini-CLI | AI Agents | AI agentReported | Not documented | Your choice |
| Google-GeminiNotebook | AI Assistant Fetchers | AI assistantOfficial docs | Not for user-requested fetches | Allow |
| Google-InspectionTool | Search Engine Crawlers | Product featureOfficial docs | Yes | Allow |
| Google-NotebookLM | AI Assistant Fetchers | AI assistantOfficial docs | Not for user-requested fetches | Allow |
| Google-Pinpoint | AI Assistant Fetchers | AI assistantProduct featureOfficial docs | Not for user-requested fetches | Allow |
| Google-Read-Aloud | Ads and Platform Fetchers | Product featureOfficial docs | Not for user-requested fetches | Allow |
| Google-Safety | Ads and Platform Fetchers | Product featureOfficial docs | No | Allow |
| Google-Site-Verification | Ads and Platform Fetchers | Product featureOfficial docs | Not for user-requested fetches | Allow |
| GoogleAgent-Mariner | AI Agents | AI agentReported | Not documented | Allow |
| GoogleAgent-URLContext | AI Assistant Fetchers | AI assistantReported | Not documented | Your choice |
| Googlebot | Search Engine Crawlers | Search indexOfficial docs | Yes | Allow |
| Googlebot-Image | Search Engine Crawlers | Search indexOfficial docs | Yes | Allow |
| Googlebot-News | Search Engine Crawlers | Search indexOfficial docs | Yes | Allow |
| Googlebot-Video | Search Engine Crawlers | Search indexOfficial docs | Yes | Allow |
| GoogleMessages | Social and Messaging Previews | Link previewOfficial docs | Not for user-requested fetches | Allow |
| GoogleOther | AI Training Crawlers | AI trainingProduct featureOfficial docs | Yes | Your choice |
| GoogleOther-Image | AI Training Crawlers | AI trainingProduct featureOfficial docs | Yes | Your choice |
| GoogleOther-Video | AI Training Crawlers | AI trainingProduct featureOfficial docs | Yes | Your choice |
| GoogleProducer | Ads and Platform Fetchers | Product featureOfficial docs | Not for user-requested fetches | Allow |
| Mediapartners-Google | Ads and Platform Fetchers | AdsOfficial docs | Yes | Allow |
| Storebot-Google | Search Engine Crawlers | Search indexOfficial docs | Yes | Allow |
Block or allow all Google bots
The rules below name every Google token in this directory. To block training but keep AI search, delete the search and assistant tokens from the block rule.
robots.txt: block
# Block all Google bots
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: APIs-Google
User-agent: Chrome-Lighthouse
User-agent: FeedFetcher-Google
User-agent: Gemini-Deep-Research
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: Google-CWS
User-agent: Google-Extended
User-agent: Google-Firebase
User-agent: Google-Gemini-CLI
User-agent: Google-GeminiNotebook
User-agent: Google-InspectionTool
User-agent: Google-NotebookLM
User-agent: Google-Pinpoint
User-agent: Google-Read-Aloud
User-agent: Google-Safety
User-agent: Google-Site-Verification
User-agent: GoogleAgent-Mariner
User-agent: GoogleAgent-URLContext
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: GoogleMessages
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GoogleProducer
User-agent: Mediapartners-Google
User-agent: Storebot-Google
Disallow: /robots.txt: allow
# Allow all Google bots
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: APIs-Google
User-agent: Chrome-Lighthouse
User-agent: FeedFetcher-Google
User-agent: Gemini-Deep-Research
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: Google-CWS
User-agent: Google-Extended
User-agent: Google-Firebase
User-agent: Google-Gemini-CLI
User-agent: Google-GeminiNotebook
User-agent: Google-InspectionTool
User-agent: Google-NotebookLM
User-agent: Google-Pinpoint
User-agent: Google-Read-Aloud
User-agent: Google-Safety
User-agent: Google-Site-Verification
User-agent: GoogleAgent-Mariner
User-agent: GoogleAgent-URLContext
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: GoogleMessages
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GoogleProducer
User-agent: Mediapartners-Google
User-agent: Storebot-Google
Allow: /robots.txt: partial
# Keep all Google bots out of private areas only
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: APIs-Google
User-agent: Chrome-Lighthouse
User-agent: FeedFetcher-Google
User-agent: Gemini-Deep-Research
User-agent: Google-Agent
User-agent: Google-CloudVertexBot
User-agent: Google-CWS
User-agent: Google-Extended
User-agent: Google-Firebase
User-agent: Google-Gemini-CLI
User-agent: Google-GeminiNotebook
User-agent: Google-InspectionTool
User-agent: Google-NotebookLM
User-agent: Google-Pinpoint
User-agent: Google-Read-Aloud
User-agent: Google-Safety
User-agent: Google-Site-Verification
User-agent: GoogleAgent-Mariner
User-agent: GoogleAgent-URLContext
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: GoogleMessages
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GoogleProducer
User-agent: Mediapartners-Google
User-agent: Storebot-Google
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to all Google bots
if ($http_user_agent ~* "(AdsBot\-Google|AdsBot\-Google\-Mobile|APIs\-Google|Chrome\-Lighthouse|FeedFetcher\-Google|Gemini\-Deep\-Research|Google\-Agent|Google\-CloudVertexBot|Google\-CWS|Google\-Extended|Google\-Firebase|Google\-Gemini\-CLI|Google\-GeminiNotebook|Google\-InspectionTool|Google\-NotebookLM|Google\-Pinpoint|Google\-Read\-Aloud|Google\-Safety|Google\-Site\-Verification|GoogleAgent\-Mariner|GoogleAgent\-URLContext|Googlebot|Googlebot\-Image|Googlebot\-News|Googlebot\-Video|GoogleMessages|GoogleOther|GoogleOther\-Image|GoogleOther\-Video|GoogleProducer|Mediapartners\-Google|Storebot\-Google)") {
return 403;
}Apache
# Apache .htaccess: return 403 to all Google bots
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AdsBot\-Google|AdsBot\-Google\-Mobile|APIs\-Google|Chrome\-Lighthouse|FeedFetcher\-Google|Gemini\-Deep\-Research|Google\-Agent|Google\-CloudVertexBot|Google\-CWS|Google\-Extended|Google\-Firebase|Google\-Gemini\-CLI|Google\-GeminiNotebook|Google\-InspectionTool|Google\-NotebookLM|Google\-Pinpoint|Google\-Read\-Aloud|Google\-Safety|Google\-Site\-Verification|GoogleAgent\-Mariner|GoogleAgent\-URLContext|Googlebot|Googlebot\-Image|Googlebot\-News|Googlebot\-Video|GoogleMessages|GoogleOther|GoogleOther\-Image|GoogleOther\-Video|GoogleProducer|Mediapartners\-Google|Storebot\-Google) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "AdsBot-Google") or (http.user_agent contains "AdsBot-Google-Mobile") or (http.user_agent contains "APIs-Google") or (http.user_agent contains "Chrome-Lighthouse") or (http.user_agent contains "FeedFetcher-Google") or (http.user_agent contains "Gemini-Deep-Research") or (http.user_agent contains "Google-Agent") or (http.user_agent contains "Google-CloudVertexBot") or (http.user_agent contains "Google-CWS") or (http.user_agent contains "Google-Extended") or (http.user_agent contains "Google-Firebase") or (http.user_agent contains "Google-Gemini-CLI") or (http.user_agent contains "Google-GeminiNotebook") or (http.user_agent contains "Google-InspectionTool") or (http.user_agent contains "Google-NotebookLM") or (http.user_agent contains "Google-Pinpoint") or (http.user_agent contains "Google-Read-Aloud") or (http.user_agent contains "Google-Safety") or (http.user_agent contains "Google-Site-Verification") or (http.user_agent contains "GoogleAgent-Mariner") or (http.user_agent contains "GoogleAgent-URLContext") or (http.user_agent contains "Googlebot") or (http.user_agent contains "Googlebot-Image") or (http.user_agent contains "Googlebot-News") or (http.user_agent contains "Googlebot-Video") or (http.user_agent contains "GoogleMessages") or (http.user_agent contains "GoogleOther") or (http.user_agent contains "GoogleOther-Image") or (http.user_agent contains "GoogleOther-Video") or (http.user_agent contains "GoogleProducer") or (http.user_agent contains "Mediapartners-Google") or (http.user_agent contains "Storebot-Google")See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
