AhrefsBot user agent
Operated by Ahrefs
AhrefsBot is Ahrefs' main crawler, which builds the backlink index behind Ahrefs and the Yep search engine.
The AhrefsBot bot at a glance
- AhrefsBot is operated by Ahrefs and crawls web pages for a search engine index, and fetches pages to power a specific product feature.
- The robots.txt token for AhrefsBot is AhrefsBot.
- Blocking an SEO tool crawler has no effect on Google or Bing rankings.
- AhrefsBot robots.txt support: Yes.
- Recommendation for Generative Engine Optimization (GEO): Your choice.
What is AhrefsBot?
AhrefsBot is one of the most active crawlers on the web after the big search engines. Its data shows your backlinks and pages to anyone using Ahrefs, including competitors.
Ahrefs documents both AhrefsBot and AhrefsSiteAudit on one page, with verification details.
Should you block AhrefsBot?
Blocking AhrefsBot is a business decision: it protects your content but can lower how well AI products know your brand.
| Effect on search rankings | Blocking an SEO tool crawler has no effect on Google or Bing rankings. |
|---|---|
| Effect on AI visibility | No effect on AI answers. Blocking hides your pages and backlinks from that SEO tool, including from competitors who use it, and breaks your own audits in that tool. |
| Follows robots.txt | Yes |
| Recommendation | Your choice |
How to block or allow AhrefsBot
Put the robots.txt rule in the file at the root of every domain and subdomain, for example https://example.com/robots.txt. The token AhrefsBot is matched case-insensitively.
robots.txt: block
# Block AhrefsBot
User-agent: AhrefsBot
Disallow: /robots.txt: allow
# Allow AhrefsBot
User-agent: AhrefsBot
Allow: /robots.txt: partial
# Keep AhrefsBot out of private areas only
User-agent: AhrefsBot
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to AhrefsBot
if ($http_user_agent ~* "(AhrefsBot)") {
return 403;
}Apache
# Apache .htaccess: return 403 to AhrefsBot
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AhrefsBot) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "AhrefsBot")How to verify real AhrefsBot traffic
Ahrefs publishes no IP list or reverse DNS pattern for AhrefsBot, so the user agent cannot be verified. Anyone can send the same string. Rate-limit unusual volume instead of trusting the name.
How to find AhrefsBot in your server logs
Search the web server access log for the token. The command below counts AhrefsBot requests per URL on Nginx or Apache.
grep -i "AhrefsBot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20For a full report per AI bot, upload the log to the AI Log File Analyzer, a free tool by Justin Hà.
AhrefsBot FAQ
AhrefsBot uses the robots.txt token "AhrefsBot". Ahrefs does not publish a full user-agent string, so match the token inside the User-Agent header.
Yes. Ahrefs states that AhrefsBot follows robots.txt rules written for the "AhrefsBot" token.
Blocking an SEO tool crawler has no effect on Google or Bing rankings. No effect on AI answers. Blocking hides your pages and backlinks from that SEO tool, including from competitors who use it, and breaks your own audits in that tool.
Add two lines to the robots.txt file at the root of each domain and subdomain: "User-agent: AhrefsBot" followed by "Disallow: /". The token is case-insensitive. Changes can take up to 24 hours to be picked up.
Not reliably. Ahrefs publishes no IP list or reverse DNS pattern for AhrefsBot, so anyone can send the same user agent. Rate-limit unusual volume instead of trusting the name.
Search your server access log for "AhrefsBot", for example with: grep -i "AhrefsBot" access.log. The AI Log File Analyzer by Justin Hà counts visits per AI bot from an uploaded log file.
Sources: Ahrefs crawler documentation. Last reviewed by Justin Hà.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
