Bot Directory
Search Engine Crawlers
Search engine crawlers build the indexes behind Google, Bing, Yandex, Baidu, Naver, Coc Coc and other search engines. Blocking one removes your pages from that engine, and AI features built on that index, such as Google AI Overviews, lose access too.
Search Engine Crawlers at a glance
- This group lists 60 bots.
- Blocking a search engine crawler removes your pages from that search engine's results.
- AI features built on that search index, such as Google AI Overviews and AI Mode on top of Googlebot, also lose access to your pages.
All Search Engine Crawlers
| Bot | Company | Purpose | robots.txt | For GEO |
|---|---|---|---|---|
| 360Spider | Qihoo 360 | Search indexReported | Not documented | Allow |
| Applebot | Apple | AI trainingAI searchSearch indexOfficial docs | Yes | Allow |
| Baiduspider | Baidu | Search indexOfficial docs | Yes | Allow |
| Baiduspider-favo | Baidu | Search indexOfficial docs | Yes | Allow |
| Baiduspider-image | Baidu | Search indexOfficial docs | Yes | Allow |
| Baiduspider-news | Baidu | Search indexOfficial docs | Yes | Allow |
| Baiduspider-video | Baidu | Search indexOfficial docs | Yes | Allow |
| Bingbot | Microsoft | Search indexOfficial docs | Yes | Allow |
| BingPreview | Microsoft | Search indexOfficial docs | Yes | Allow |
| Bravebot | Brave | AI searchSearch indexReported | Yes | Allow |
| coccocbot-image | Coc Coc | Search indexOfficial docs | Yes | Allow |
| coccocbot-web | Coc Coc | Search indexOfficial docs | Yes | Allow |
| Daum | Kakao | Search indexReported | Not documented | Allow |
| DuckDuckBot | DuckDuckGo | Search indexOfficial docs | Yes | Allow |
| Google-InspectionTool | Product featureOfficial docs | Yes | Allow | |
| Googlebot | Search indexOfficial docs | Yes | Allow | |
| Googlebot-Image | Search indexOfficial docs | Yes | Allow | |
| Googlebot-News | Search indexOfficial docs | Yes | Allow | |
| Googlebot-Video | Search indexOfficial docs | Yes | Allow | |
| Kagibot | Kagi | Search indexOfficial docs | Yes | Allow |
| Linespider | LY Corporation (LINE) | Search indexOfficial docs | Yes | Allow |
| Mail.RU_Bot | VK | Search indexReported | Not documented | Allow |
| MojeekBot | Mojeek | Search indexOfficial docs | Yes | Allow |
| msnbot | Microsoft | Search indexReported | Not documented | Allow |
| PetalBot | Huawei | AI searchSearch indexOfficial docs | Yes | Allow |
| Qwantbot | Qwant | Search indexOfficial docs | Yes | Allow |
| SeekportBot | Seekport | Search indexOfficial docs | Yes | Allow |
| SeznamBot | Seznam | Search indexOfficial docs | Yes | Allow |
| Sogou web spider | Sogou | Search indexOfficial docs | Yes | Allow |
| Storebot-Google | Search indexOfficial docs | Yes | Allow | |
| ToutiaoSpider | ByteDance | Search indexReported | Not documented | Allow |
| Yahoo! JAPAN crawlerY!J | Yahoo! JAPAN | Search indexReported | Not documented | Allow |
| Yahoo! SlurpSlurp | Yahoo | Search indexOfficial docs | Yes | Allow |
| YandexAccessibilityBot | Yandex | Product featureOfficial docs | Yes | Allow |
| YandexBlogs | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexBot | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexCheckBot | Yandex | Product featureOfficial docs | Not for user-requested fetches | Allow |
| YandexComBot | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexFavicons | Yandex | Search indexOfficial docs | Not for user-requested fetches | Allow |
| YandexImageResizer | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexImages | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexMarket | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexMedia | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexMobileBot | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexMobileScreenShotBot | Yandex | Product featureOfficial docs | Not for user-requested fetches | Allow |
| YandexOntoDB | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexOntoDBAPI | Yandex | Search indexOfficial docs | Not for user-requested fetches | Allow |
| YandexPagechecker | Yandex | Product featureOfficial docs | Yes | Allow |
| YandexRCA | Yandex | Link previewOfficial docs | Not for user-requested fetches | Allow |
| YandexRenderResourcesBot | Yandex | Search indexOfficial docs | Not for user-requested fetches | Allow |
| YandexScreenshotBot | Yandex | Product featureOfficial docs | Not for user-requested fetches | Allow |
| YandexSitelinks | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexSpravBot | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexVerticals | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexVertis | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexVideo | Yandex | Search indexOfficial docs | Yes | Allow |
| YandexVideoParser | Yandex | Search indexOfficial docs | Not for user-requested fetches | Allow |
| YandexWebmaster | Yandex | Product featureOfficial docs | Yes | Allow |
| Yeti | Naver | Search indexOfficial docs | Yes | Allow |
| YisouSpider | Alibaba | Search indexReported | Not documented | Allow |
Block or allow every bot in Search Engine Crawlers at once
The rules below name all 60 robots.txt tokens in this group. Remove the lines for any bot you want to treat differently.
robots.txt: block
# Block all Search Engine Crawlers
User-agent: 360Spider
User-agent: Applebot
User-agent: Baiduspider
User-agent: Baiduspider-favo
User-agent: Baiduspider-image
User-agent: Baiduspider-news
User-agent: Baiduspider-video
User-agent: bingbot
User-agent: BingPreview
User-agent: Bravebot
User-agent: coccocbot-image
User-agent: coccocbot-web
User-agent: Daum
User-agent: DuckDuckBot
User-agent: Google-InspectionTool
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: Kagibot
User-agent: Linespider
User-agent: Mail.RU_Bot
User-agent: MojeekBot
User-agent: msnbot
User-agent: PetalBot
User-agent: Qwantbot
User-agent: SeekportBot
User-agent: SeznamBot
User-agent: Sogou web spider
User-agent: Storebot-Google
User-agent: ToutiaoSpider
User-agent: Y!J
User-agent: Slurp
User-agent: YandexAccessibilityBot
User-agent: YandexBlogs
User-agent: YandexBot
User-agent: YandexCheckBot
User-agent: YandexComBot
User-agent: YandexFavicons
User-agent: YandexImageResizer
User-agent: YandexImages
User-agent: YandexMarket
User-agent: YandexMedia
User-agent: YandexMobileBot
User-agent: YandexMobileScreenShotBot
User-agent: YandexOntoDB
User-agent: YandexOntoDBAPI
User-agent: YandexPagechecker
User-agent: YandexRCA
User-agent: YandexRenderResourcesBot
User-agent: YandexScreenshotBot
User-agent: YandexSitelinks
User-agent: YandexSpravBot
User-agent: YandexVerticals
User-agent: YandexVertis
User-agent: YandexVideo
User-agent: YandexVideoParser
User-agent: YandexWebmaster
User-agent: Yeti
User-agent: YisouSpider
Disallow: /robots.txt: allow
# Allow all Search Engine Crawlers
User-agent: 360Spider
User-agent: Applebot
User-agent: Baiduspider
User-agent: Baiduspider-favo
User-agent: Baiduspider-image
User-agent: Baiduspider-news
User-agent: Baiduspider-video
User-agent: bingbot
User-agent: BingPreview
User-agent: Bravebot
User-agent: coccocbot-image
User-agent: coccocbot-web
User-agent: Daum
User-agent: DuckDuckBot
User-agent: Google-InspectionTool
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: Kagibot
User-agent: Linespider
User-agent: Mail.RU_Bot
User-agent: MojeekBot
User-agent: msnbot
User-agent: PetalBot
User-agent: Qwantbot
User-agent: SeekportBot
User-agent: SeznamBot
User-agent: Sogou web spider
User-agent: Storebot-Google
User-agent: ToutiaoSpider
User-agent: Y!J
User-agent: Slurp
User-agent: YandexAccessibilityBot
User-agent: YandexBlogs
User-agent: YandexBot
User-agent: YandexCheckBot
User-agent: YandexComBot
User-agent: YandexFavicons
User-agent: YandexImageResizer
User-agent: YandexImages
User-agent: YandexMarket
User-agent: YandexMedia
User-agent: YandexMobileBot
User-agent: YandexMobileScreenShotBot
User-agent: YandexOntoDB
User-agent: YandexOntoDBAPI
User-agent: YandexPagechecker
User-agent: YandexRCA
User-agent: YandexRenderResourcesBot
User-agent: YandexScreenshotBot
User-agent: YandexSitelinks
User-agent: YandexSpravBot
User-agent: YandexVerticals
User-agent: YandexVertis
User-agent: YandexVideo
User-agent: YandexVideoParser
User-agent: YandexWebmaster
User-agent: Yeti
User-agent: YisouSpider
Allow: /robots.txt: partial
# Keep all Search Engine Crawlers out of private areas only
User-agent: 360Spider
User-agent: Applebot
User-agent: Baiduspider
User-agent: Baiduspider-favo
User-agent: Baiduspider-image
User-agent: Baiduspider-news
User-agent: Baiduspider-video
User-agent: bingbot
User-agent: BingPreview
User-agent: Bravebot
User-agent: coccocbot-image
User-agent: coccocbot-web
User-agent: Daum
User-agent: DuckDuckBot
User-agent: Google-InspectionTool
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-News
User-agent: Googlebot-Video
User-agent: Kagibot
User-agent: Linespider
User-agent: Mail.RU_Bot
User-agent: MojeekBot
User-agent: msnbot
User-agent: PetalBot
User-agent: Qwantbot
User-agent: SeekportBot
User-agent: SeznamBot
User-agent: Sogou web spider
User-agent: Storebot-Google
User-agent: ToutiaoSpider
User-agent: Y!J
User-agent: Slurp
User-agent: YandexAccessibilityBot
User-agent: YandexBlogs
User-agent: YandexBot
User-agent: YandexCheckBot
User-agent: YandexComBot
User-agent: YandexFavicons
User-agent: YandexImageResizer
User-agent: YandexImages
User-agent: YandexMarket
User-agent: YandexMedia
User-agent: YandexMobileBot
User-agent: YandexMobileScreenShotBot
User-agent: YandexOntoDB
User-agent: YandexOntoDBAPI
User-agent: YandexPagechecker
User-agent: YandexRCA
User-agent: YandexRenderResourcesBot
User-agent: YandexScreenshotBot
User-agent: YandexSitelinks
User-agent: YandexSpravBot
User-agent: YandexVerticals
User-agent: YandexVertis
User-agent: YandexVideo
User-agent: YandexVideoParser
User-agent: YandexWebmaster
User-agent: Yeti
User-agent: YisouSpider
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to all Search Engine Crawlers
if ($http_user_agent ~* "(360Spider|Applebot|Baiduspider|Baiduspider\-favo|Baiduspider\-image|Baiduspider\-news|Baiduspider\-video|bingbot|BingPreview|Bravebot|coccocbot\-image|coccocbot\-web|Daum|DuckDuckBot|Google\-InspectionTool|Googlebot|Googlebot\-Image|Googlebot\-News|Googlebot\-Video|Kagibot|Linespider|Mail\.RU_Bot|MojeekBot|msnbot|PetalBot|Qwantbot|SeekportBot|SeznamBot|Sogou web spider|Storebot\-Google|ToutiaoSpider|Y\!J|Slurp|YandexAccessibilityBot|YandexBlogs|YandexBot|YandexCheckBot|YandexComBot|YandexFavicons|YandexImageResizer|YandexImages|YandexMarket|YandexMedia|YandexMobileBot|YandexMobileScreenShotBot|YandexOntoDB|YandexOntoDBAPI|YandexPagechecker|YandexRCA|YandexRenderResourcesBot|YandexScreenshotBot|YandexSitelinks|YandexSpravBot|YandexVerticals|YandexVertis|YandexVideo|YandexVideoParser|YandexWebmaster|Yeti|YisouSpider)") {
return 403;
}Apache
# Apache .htaccess: return 403 to all Search Engine Crawlers
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (360Spider|Applebot|Baiduspider|Baiduspider\-favo|Baiduspider\-image|Baiduspider\-news|Baiduspider\-video|bingbot|BingPreview|Bravebot|coccocbot\-image|coccocbot\-web|Daum|DuckDuckBot|Google\-InspectionTool|Googlebot|Googlebot\-Image|Googlebot\-News|Googlebot\-Video|Kagibot|Linespider|Mail\.RU_Bot|MojeekBot|msnbot|PetalBot|Qwantbot|SeekportBot|SeznamBot|Sogou web spider|Storebot\-Google|ToutiaoSpider|Y\!J|Slurp|YandexAccessibilityBot|YandexBlogs|YandexBot|YandexCheckBot|YandexComBot|YandexFavicons|YandexImageResizer|YandexImages|YandexMarket|YandexMedia|YandexMobileBot|YandexMobileScreenShotBot|YandexOntoDB|YandexOntoDBAPI|YandexPagechecker|YandexRCA|YandexRenderResourcesBot|YandexScreenshotBot|YandexSitelinks|YandexSpravBot|YandexVerticals|YandexVertis|YandexVideo|YandexVideoParser|YandexWebmaster|Yeti|YisouSpider) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "360Spider") or (http.user_agent contains "Applebot") or (http.user_agent contains "Baiduspider") or (http.user_agent contains "Baiduspider-favo") or (http.user_agent contains "Baiduspider-image") or (http.user_agent contains "Baiduspider-news") or (http.user_agent contains "Baiduspider-video") or (http.user_agent contains "bingbot") or (http.user_agent contains "BingPreview") or (http.user_agent contains "Bravebot") or (http.user_agent contains "coccocbot-image") or (http.user_agent contains "coccocbot-web") or (http.user_agent contains "Daum") or (http.user_agent contains "DuckDuckBot") or (http.user_agent contains "Google-InspectionTool") or (http.user_agent contains "Googlebot") or (http.user_agent contains "Googlebot-Image") or (http.user_agent contains "Googlebot-News") or (http.user_agent contains "Googlebot-Video") or (http.user_agent contains "Kagibot") or (http.user_agent contains "Linespider") or (http.user_agent contains "Mail.RU_Bot") or (http.user_agent contains "MojeekBot") or (http.user_agent contains "msnbot") or (http.user_agent contains "PetalBot") or (http.user_agent contains "Qwantbot") or (http.user_agent contains "SeekportBot") or (http.user_agent contains "SeznamBot") or (http.user_agent contains "Sogou web spider") or (http.user_agent contains "Storebot-Google") or (http.user_agent contains "ToutiaoSpider") or (http.user_agent contains "Y!J") or (http.user_agent contains "Slurp") or (http.user_agent contains "YandexAccessibilityBot") or (http.user_agent contains "YandexBlogs") or (http.user_agent contains "YandexBot") or (http.user_agent contains "YandexCheckBot") or (http.user_agent contains "YandexComBot") or (http.user_agent contains "YandexFavicons") or (http.user_agent contains "YandexImageResizer") or (http.user_agent contains "YandexImages") or (http.user_agent contains "YandexMarket") or (http.user_agent contains "YandexMedia") or (http.user_agent contains "YandexMobileBot") or (http.user_agent contains "YandexMobileScreenShotBot") or (http.user_agent contains "YandexOntoDB") or (http.user_agent contains "YandexOntoDBAPI") or (http.user_agent contains "YandexPagechecker") or (http.user_agent contains "YandexRCA") or (http.user_agent contains "YandexRenderResourcesBot") or (http.user_agent contains "YandexScreenshotBot") or (http.user_agent contains "YandexSitelinks") or (http.user_agent contains "YandexSpravBot") or (http.user_agent contains "YandexVerticals") or (http.user_agent contains "YandexVertis") or (http.user_agent contains "YandexVideo") or (http.user_agent contains "YandexVideoParser") or (http.user_agent contains "YandexWebmaster") or (http.user_agent contains "Yeti") or (http.user_agent contains "YisouSpider")Other bot groups
AI Training Crawlers
44 bots
Collect public pages that may be used to train AI models. Blocking them has no effect on Google or Bing rankings.
AI Search Crawlers
21 bots
Index pages so AI search products can show and cite them. Blocking them removes you from those answers.
AI Assistant Fetchers
30 bots
Open a page when a user asks an AI assistant about it. They act for one person, not for an index.
AI Agents
14 bots
Use a browser to complete tasks for users, such as comparing, booking or buying.
Ads and Platform Fetchers
38 bots
Serve one platform feature: ad checks, feeds, verification or safety scans.
SEO Tool Crawlers
29 bots
Crawl the web for SEO platforms: backlink indexes, rank data and site audits. Blocking them hides your site from those tools.
Archives and Research Crawlers
4 bots
Save public pages for web archives, libraries and research. Blocking them stops new snapshots of your site.
Social and Messaging Previews
22 bots
Build the preview card when your link is shared in a social or messaging app. Blocking them breaks those cards.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
