Bot Directory
Ads and Platform Fetchers
Platform fetchers visit a page for one product feature: checking an ad landing page, reading a feed, verifying site ownership or scanning for abuse. They usually follow a user or advertiser action rather than crawling the web.
Ads and Platform Fetchers at a glance
- This group lists 38 bots.
- Blocking a platform fetcher has no effect on organic search rankings.
- Blocking breaks the feature the fetcher serves, such as ad approval or feed updates.
All Ads and Platform Fetchers
| Bot | Company | Purpose | robots.txt | For GEO |
|---|---|---|---|---|
| AdIdxBot | Microsoft | AdsOfficial docs | Yes | Allow |
| AdsBot-Google | AdsOfficial docs | Yes | Allow | |
| AdsBot-Google-Mobile | AdsOfficial docs | Yes | Allow | |
| AmazonAdBot | Amazon | AdsReported | Not documented | Allow |
| APIs-Google | Product featureOfficial docs | Yes | Allow | |
| Baiduspider-ads | Baidu | AdsOfficial docs | Yes | Allow |
| Baiduspider-cpro | Baidu | AdsOfficial docs | Yes | Allow |
| CriteoBot | Criteo | AdsReported | Not documented | Allow |
| FeedFetcher-Google | Product featureOfficial docs | Not for user-requested fetches | Allow | |
| Google-CWS | Product featureOfficial docs | Not for user-requested fetches | Allow | |
| Google-Read-Aloud | Product featureOfficial docs | Not for user-requested fetches | Allow | |
| Google-Safety | Product featureOfficial docs | No | Allow | |
| Google-Site-Verification | Product featureOfficial docs | Not for user-requested fetches | Allow | |
| GoogleProducer | Product featureOfficial docs | Not for user-requested fetches | Allow | |
| GrapeshotCrawler | Oracle | AdsReported | Not documented | Your choice |
| IAS crawlerias_crawler | Integral Ad Science | AdsReported | Not documented | Allow |
| Leikibot | DoubleVerify | AdsReported | Not documented | Your choice |
| Mediapartners-Google | AdsOfficial docs | Yes | Allow | |
| Meta-ExternalAds | Meta | AdsOfficial docs | Yes | Allow |
| MgidBot | MGID | AdsReported | Not documented | Your choice |
| OAI-AdsBot | OpenAI | AdsOfficial docs | Not documented | Allow |
| Outbrain | Outbrain | AdsReported | Not documented | Allow |
| proximic | Comscore | AdsReported | Not documented | Your choice |
| Quantcastbot | Quantcast | AdsReported | Not documented | Your choice |
| SnapchatAds | Snap | AdsReported | Not documented | Allow |
| Taboolabot | Taboola | AdsReported | Not documented | Allow |
| TTD-Content | The Trade Desk | AdsReported | Not documented | Your choice |
| YaDirectFetcher | Yandex | AdsOfficial docs | No | Allow |
| YandexAdNet | Yandex | AdsOfficial docs | Yes | Allow |
| YandexDirect | Yandex | AdsOfficial docs | Not for user-requested fetches | Allow |
| YandexDirectDyn | Yandex | AdsOfficial docs | Not for user-requested fetches | Allow |
| YandexFeed | Yandex | Product featureOfficial docs | Not for user-requested fetches | Allow |
| YandexMetrika | Yandex | Product featureOfficial docs | No | Allow |
| YandexPartner | Yandex | AdsOfficial docs | Not for user-requested fetches | Allow |
| YandexSearchShop | Yandex | Product featureOfficial docs | Not for user-requested fetches | Allow |
| YandexTracker | Yandex | Product featureOfficial docs | Not for user-requested fetches | Allow |
| YandexUserproxy | Yandex | Product featureOfficial docs | Not for user-requested fetches | Allow |
| YextBot | Yext | Product featureReported | Not documented | Allow |
Block or allow every bot in Ads and Platform Fetchers at once
The rules below name all 38 robots.txt tokens in this group. Remove the lines for any bot you want to treat differently.
robots.txt: block
# Block all Ads and Platform Fetchers
User-agent: adidxbot
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AmazonAdBot
User-agent: APIs-Google
User-agent: Baiduspider-ads
User-agent: Baiduspider-cpro
User-agent: CriteoBot
User-agent: FeedFetcher-Google
User-agent: Google-CWS
User-agent: Google-Read-Aloud
User-agent: Google-Safety
User-agent: Google-Site-Verification
User-agent: GoogleProducer
User-agent: GrapeshotCrawler
User-agent: ias_crawler
User-agent: Leikibot
User-agent: Mediapartners-Google
User-agent: meta-externalads
User-agent: MgidBot
User-agent: OAI-AdsBot
User-agent: Outbrain
User-agent: proximic
User-agent: Quantcastbot
User-agent: SnapchatAds
User-agent: Taboolabot
User-agent: TTD-Content
User-agent: YaDirectFetcher
User-agent: YandexAdNet
User-agent: YandexDirect
User-agent: YandexDirectDyn
User-agent: YandexFeed
User-agent: YandexMetrika
User-agent: YandexPartner
User-agent: YandexSearchShop
User-agent: YandexTracker
User-agent: YandexUserproxy
User-agent: YextBot
Disallow: /robots.txt: allow
# Allow all Ads and Platform Fetchers
User-agent: adidxbot
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AmazonAdBot
User-agent: APIs-Google
User-agent: Baiduspider-ads
User-agent: Baiduspider-cpro
User-agent: CriteoBot
User-agent: FeedFetcher-Google
User-agent: Google-CWS
User-agent: Google-Read-Aloud
User-agent: Google-Safety
User-agent: Google-Site-Verification
User-agent: GoogleProducer
User-agent: GrapeshotCrawler
User-agent: ias_crawler
User-agent: Leikibot
User-agent: Mediapartners-Google
User-agent: meta-externalads
User-agent: MgidBot
User-agent: OAI-AdsBot
User-agent: Outbrain
User-agent: proximic
User-agent: Quantcastbot
User-agent: SnapchatAds
User-agent: Taboolabot
User-agent: TTD-Content
User-agent: YaDirectFetcher
User-agent: YandexAdNet
User-agent: YandexDirect
User-agent: YandexDirectDyn
User-agent: YandexFeed
User-agent: YandexMetrika
User-agent: YandexPartner
User-agent: YandexSearchShop
User-agent: YandexTracker
User-agent: YandexUserproxy
User-agent: YextBot
Allow: /robots.txt: partial
# Keep all Ads and Platform Fetchers out of private areas only
User-agent: adidxbot
User-agent: AdsBot-Google
User-agent: AdsBot-Google-Mobile
User-agent: AmazonAdBot
User-agent: APIs-Google
User-agent: Baiduspider-ads
User-agent: Baiduspider-cpro
User-agent: CriteoBot
User-agent: FeedFetcher-Google
User-agent: Google-CWS
User-agent: Google-Read-Aloud
User-agent: Google-Safety
User-agent: Google-Site-Verification
User-agent: GoogleProducer
User-agent: GrapeshotCrawler
User-agent: ias_crawler
User-agent: Leikibot
User-agent: Mediapartners-Google
User-agent: meta-externalads
User-agent: MgidBot
User-agent: OAI-AdsBot
User-agent: Outbrain
User-agent: proximic
User-agent: Quantcastbot
User-agent: SnapchatAds
User-agent: Taboolabot
User-agent: TTD-Content
User-agent: YaDirectFetcher
User-agent: YandexAdNet
User-agent: YandexDirect
User-agent: YandexDirectDyn
User-agent: YandexFeed
User-agent: YandexMetrika
User-agent: YandexPartner
User-agent: YandexSearchShop
User-agent: YandexTracker
User-agent: YandexUserproxy
User-agent: YextBot
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to all Ads and Platform Fetchers
if ($http_user_agent ~* "(adidxbot|AdsBot\-Google|AdsBot\-Google\-Mobile|AmazonAdBot|APIs\-Google|Baiduspider\-ads|Baiduspider\-cpro|CriteoBot|FeedFetcher\-Google|Google\-CWS|Google\-Read\-Aloud|Google\-Safety|Google\-Site\-Verification|GoogleProducer|GrapeshotCrawler|ias_crawler|Leikibot|Mediapartners\-Google|meta\-externalads|MgidBot|OAI\-AdsBot|Outbrain|proximic|Quantcastbot|SnapchatAds|Taboolabot|TTD\-Content|YaDirectFetcher|YandexAdNet|YandexDirect|YandexDirectDyn|YandexFeed|YandexMetrika|YandexPartner|YandexSearchShop|YandexTracker|YandexUserproxy|YextBot)") {
return 403;
}Apache
# Apache .htaccess: return 403 to all Ads and Platform Fetchers
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (adidxbot|AdsBot\-Google|AdsBot\-Google\-Mobile|AmazonAdBot|APIs\-Google|Baiduspider\-ads|Baiduspider\-cpro|CriteoBot|FeedFetcher\-Google|Google\-CWS|Google\-Read\-Aloud|Google\-Safety|Google\-Site\-Verification|GoogleProducer|GrapeshotCrawler|ias_crawler|Leikibot|Mediapartners\-Google|meta\-externalads|MgidBot|OAI\-AdsBot|Outbrain|proximic|Quantcastbot|SnapchatAds|Taboolabot|TTD\-Content|YaDirectFetcher|YandexAdNet|YandexDirect|YandexDirectDyn|YandexFeed|YandexMetrika|YandexPartner|YandexSearchShop|YandexTracker|YandexUserproxy|YextBot) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "adidxbot") or (http.user_agent contains "AdsBot-Google") or (http.user_agent contains "AdsBot-Google-Mobile") or (http.user_agent contains "AmazonAdBot") or (http.user_agent contains "APIs-Google") or (http.user_agent contains "Baiduspider-ads") or (http.user_agent contains "Baiduspider-cpro") or (http.user_agent contains "CriteoBot") or (http.user_agent contains "FeedFetcher-Google") or (http.user_agent contains "Google-CWS") or (http.user_agent contains "Google-Read-Aloud") or (http.user_agent contains "Google-Safety") or (http.user_agent contains "Google-Site-Verification") or (http.user_agent contains "GoogleProducer") or (http.user_agent contains "GrapeshotCrawler") or (http.user_agent contains "ias_crawler") or (http.user_agent contains "Leikibot") or (http.user_agent contains "Mediapartners-Google") or (http.user_agent contains "meta-externalads") or (http.user_agent contains "MgidBot") or (http.user_agent contains "OAI-AdsBot") or (http.user_agent contains "Outbrain") or (http.user_agent contains "proximic") or (http.user_agent contains "Quantcastbot") or (http.user_agent contains "SnapchatAds") or (http.user_agent contains "Taboolabot") or (http.user_agent contains "TTD-Content") or (http.user_agent contains "YaDirectFetcher") or (http.user_agent contains "YandexAdNet") or (http.user_agent contains "YandexDirect") or (http.user_agent contains "YandexDirectDyn") or (http.user_agent contains "YandexFeed") or (http.user_agent contains "YandexMetrika") or (http.user_agent contains "YandexPartner") or (http.user_agent contains "YandexSearchShop") or (http.user_agent contains "YandexTracker") or (http.user_agent contains "YandexUserproxy") or (http.user_agent contains "YextBot")Other bot groups
AI Training Crawlers
44 bots
Collect public pages that may be used to train AI models. Blocking them has no effect on Google or Bing rankings.
AI Search Crawlers
21 bots
Index pages so AI search products can show and cite them. Blocking them removes you from those answers.
AI Assistant Fetchers
30 bots
Open a page when a user asks an AI assistant about it. They act for one person, not for an index.
AI Agents
14 bots
Use a browser to complete tasks for users, such as comparing, booking or buying.
Search Engine Crawlers
60 bots
Index pages for classic search engines. Blocking them removes you from search results.
SEO Tool Crawlers
29 bots
Crawl the web for SEO platforms: backlink indexes, rank data and site audits. Blocking them hides your site from those tools.
Archives and Research Crawlers
4 bots
Save public pages for web archives, libraries and research. Blocking them stops new snapshots of your site.
Social and Messaging Previews
22 bots
Build the preview card when your link is shared in a social or messaging app. Blocking them breaks those cards.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
