Bot Directory
SEO Tool Crawlers
SEO tool crawlers build the backlink, keyword and site audit data behind platforms such as Ahrefs, Semrush, Moz, Majestic and Screaming Frog. They do not decide search rankings, but they decide what you and your competitors see about your site inside those tools.
SEO Tool Crawlers at a glance
- This group lists 29 bots.
- Blocking an SEO tool crawler has no effect on Google or Bing rankings.
- No effect on AI answers. Blocking hides your pages and backlinks from that SEO tool, including from competitors who use it, and breaks your own audits in that tool.
All SEO Tool Crawlers
| Bot | Company | Purpose | robots.txt | For GEO |
|---|---|---|---|---|
| AhrefsBot | Ahrefs | Search indexProduct featureOfficial docs | Yes | Your choice |
| AhrefsSiteAudit | Ahrefs | Product featureOfficial docs | Yes | Allow |
| AwarioBot | Awario | Product featureReported | Not documented | Your choice |
| Barkrowler | Babbar | Search indexProduct featureOfficial docs | Yes | Your choice |
| BLEXBot | WebMeUp | Search indexProduct featureReported | Yes | Your choice |
| Botify | Botify | Product featureReported | Not documented | Your choice |
| BrightEdge Crawler | BrightEdge | Product featureReported | Not documented | Your choice |
| Caliperbot | Conductor | Product featureReported | Not documented | Your choice |
| Chrome-Lighthouse | Product featureReported | Not documented | Allow | |
| DataForSeoBot | DataForSEO | Search indexProduct featureOfficial docs | Yes | Your choice |
| DotBot | Moz | Search indexProduct featureOfficial docs | Yes | Your choice |
| magpie-crawler | Brandwatch | Product featureReported | Not documented | Your choice |
| MJ12bot | Majestic | Search indexProduct featureOfficial docs | Yes | Your choice |
| OnCrawl | Oncrawl | Product featureReported | Not documented | Your choice |
| rogerbot | Moz | Product featureOfficial docs | Yes | Allow |
| Screaming Frog SEO Spider | Screaming Frog | Product featureOfficial docs | Yes | Your choice |
| SemrushBot | Semrush | Search indexProduct featureOfficial docs | Yes | Your choice |
| SemrushBot-BA | Semrush | Product featureOfficial docs | Yes | Your choice |
| SemrushBot-OCOB | Semrush | Product featureOfficial docs | Yes | Your choice |
| SemrushBot-SI | Semrush | Product featureOfficial docs | Yes | Your choice |
| SemrushBot-SWA | Semrush | Product featureOfficial docs | Yes | Your choice |
| SeobilityBot | Seobility | Product featureOfficial docs | Yes | Your choice |
| SERankingBacklinksBot | SE Ranking | Search indexProduct featureReported | Not documented | Your choice |
| serpstatbot | Serpstat | Search indexProduct featureOfficial docs | Yes | Your choice |
| SISTRIX CrawlerSISTRIX | SISTRIX | Product featureReported | Not documented | Your choice |
| SiteAuditBot | Semrush | Product featureOfficial docs | Yes | Allow |
| Siteimprove | Siteimprove | Product featureReported | Not documented | Your choice |
| SplitSignalBot | Semrush | Product featureOfficial docs | Yes | Your choice |
| ZoomBot | SEOZoom | Product featureReported | Not documented | Your choice |
Block or allow every bot in SEO Tool Crawlers at once
The rules below name all 29 robots.txt tokens in this group. Remove the lines for any bot you want to treat differently.
robots.txt: block
# Block all SEO Tool Crawlers
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: AwarioBot
User-agent: Barkrowler
User-agent: BLEXBot
User-agent: botify
User-agent: BrightEdge Crawler
User-agent: Caliperbot
User-agent: Chrome-Lighthouse
User-agent: DataForSeoBot
User-agent: DotBot
User-agent: magpie-crawler
User-agent: MJ12bot
User-agent: OnCrawl
User-agent: rogerbot
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: SemrushBot-BA
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: SeobilityBot
User-agent: SERankingBacklinksBot
User-agent: serpstatbot
User-agent: SISTRIX
User-agent: SiteAuditBot
User-agent: Siteimprove
User-agent: SplitSignalBot
User-agent: ZoomBot
Disallow: /robots.txt: allow
# Allow all SEO Tool Crawlers
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: AwarioBot
User-agent: Barkrowler
User-agent: BLEXBot
User-agent: botify
User-agent: BrightEdge Crawler
User-agent: Caliperbot
User-agent: Chrome-Lighthouse
User-agent: DataForSeoBot
User-agent: DotBot
User-agent: magpie-crawler
User-agent: MJ12bot
User-agent: OnCrawl
User-agent: rogerbot
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: SemrushBot-BA
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: SeobilityBot
User-agent: SERankingBacklinksBot
User-agent: serpstatbot
User-agent: SISTRIX
User-agent: SiteAuditBot
User-agent: Siteimprove
User-agent: SplitSignalBot
User-agent: ZoomBot
Allow: /robots.txt: partial
# Keep all SEO Tool Crawlers out of private areas only
User-agent: AhrefsBot
User-agent: AhrefsSiteAudit
User-agent: AwarioBot
User-agent: Barkrowler
User-agent: BLEXBot
User-agent: botify
User-agent: BrightEdge Crawler
User-agent: Caliperbot
User-agent: Chrome-Lighthouse
User-agent: DataForSeoBot
User-agent: DotBot
User-agent: magpie-crawler
User-agent: MJ12bot
User-agent: OnCrawl
User-agent: rogerbot
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: SemrushBot-BA
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: SeobilityBot
User-agent: SERankingBacklinksBot
User-agent: serpstatbot
User-agent: SISTRIX
User-agent: SiteAuditBot
User-agent: Siteimprove
User-agent: SplitSignalBot
User-agent: ZoomBot
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to all SEO Tool Crawlers
if ($http_user_agent ~* "(AhrefsBot|AhrefsSiteAudit|AwarioBot|Barkrowler|BLEXBot|botify|BrightEdge Crawler|Caliperbot|Chrome\-Lighthouse|DataForSeoBot|DotBot|magpie\-crawler|MJ12bot|OnCrawl|rogerbot|Screaming Frog SEO Spider|SemrushBot|SemrushBot\-BA|SemrushBot\-OCOB|SemrushBot\-SI|SemrushBot\-SWA|SeobilityBot|SERankingBacklinksBot|serpstatbot|SISTRIX|SiteAuditBot|Siteimprove|SplitSignalBot|ZoomBot)") {
return 403;
}Apache
# Apache .htaccess: return 403 to all SEO Tool Crawlers
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AhrefsBot|AhrefsSiteAudit|AwarioBot|Barkrowler|BLEXBot|botify|BrightEdge Crawler|Caliperbot|Chrome\-Lighthouse|DataForSeoBot|DotBot|magpie\-crawler|MJ12bot|OnCrawl|rogerbot|Screaming Frog SEO Spider|SemrushBot|SemrushBot\-BA|SemrushBot\-OCOB|SemrushBot\-SI|SemrushBot\-SWA|SeobilityBot|SERankingBacklinksBot|serpstatbot|SISTRIX|SiteAuditBot|Siteimprove|SplitSignalBot|ZoomBot) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "AhrefsBot") or (http.user_agent contains "AhrefsSiteAudit") or (http.user_agent contains "AwarioBot") or (http.user_agent contains "Barkrowler") or (http.user_agent contains "BLEXBot") or (http.user_agent contains "botify") or (http.user_agent contains "BrightEdge Crawler") or (http.user_agent contains "Caliperbot") or (http.user_agent contains "Chrome-Lighthouse") or (http.user_agent contains "DataForSeoBot") or (http.user_agent contains "DotBot") or (http.user_agent contains "magpie-crawler") or (http.user_agent contains "MJ12bot") or (http.user_agent contains "OnCrawl") or (http.user_agent contains "rogerbot") or (http.user_agent contains "Screaming Frog SEO Spider") or (http.user_agent contains "SemrushBot") or (http.user_agent contains "SemrushBot-BA") or (http.user_agent contains "SemrushBot-OCOB") or (http.user_agent contains "SemrushBot-SI") or (http.user_agent contains "SemrushBot-SWA") or (http.user_agent contains "SeobilityBot") or (http.user_agent contains "SERankingBacklinksBot") or (http.user_agent contains "serpstatbot") or (http.user_agent contains "SISTRIX") or (http.user_agent contains "SiteAuditBot") or (http.user_agent contains "Siteimprove") or (http.user_agent contains "SplitSignalBot") or (http.user_agent contains "ZoomBot")Other bot groups
AI Training Crawlers
44 bots
Collect public pages that may be used to train AI models. Blocking them has no effect on Google or Bing rankings.
AI Search Crawlers
21 bots
Index pages so AI search products can show and cite them. Blocking them removes you from those answers.
AI Assistant Fetchers
30 bots
Open a page when a user asks an AI assistant about it. They act for one person, not for an index.
AI Agents
14 bots
Use a browser to complete tasks for users, such as comparing, booking or buying.
Search Engine Crawlers
60 bots
Index pages for classic search engines. Blocking them removes you from search results.
Ads and Platform Fetchers
38 bots
Serve one platform feature: ad checks, feeds, verification or safety scans.
Archives and Research Crawlers
4 bots
Save public pages for web archives, libraries and research. Blocking them stops new snapshots of your site.
Social and Messaging Previews
22 bots
Build the preview card when your link is shared in a social or messaging app. Blocking them breaks those cards.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
