AhrefsSiteAudit user agent
Operated by Ahrefs
AhrefsSiteAudit is the crawler behind Ahrefs Site Audit, which audits sites that Ahrefs users have added and verified.
The AhrefsSiteAudit bot at a glance
- AhrefsSiteAudit is operated by Ahrefs and fetches pages to power a specific product feature.
- The robots.txt token for AhrefsSiteAudit is AhrefsSiteAudit.
- Blocking an SEO tool crawler has no effect on Google or Bing rankings.
- AhrefsSiteAudit robots.txt support: Yes.
- Recommendation for Generative Engine Optimization (GEO): Allow.
What is AhrefsSiteAudit?
AhrefsSiteAudit crawls a website when an Ahrefs user runs a technical SEO audit on it.
Allow it on your own site if you or your agency audit it with Ahrefs.
Should you block AhrefsSiteAudit?
Allow AhrefsSiteAudit unless it causes a measurable load problem.
| Effect on search rankings | Blocking an SEO tool crawler has no effect on Google or Bing rankings. |
|---|---|
| Effect on AI visibility | No effect on AI answers. Blocking hides your pages and backlinks from that SEO tool, including from competitors who use it, and breaks your own audits in that tool. |
| Follows robots.txt | Yes |
| Recommendation | Allow |
How to block or allow AhrefsSiteAudit
Put the robots.txt rule in the file at the root of every domain and subdomain, for example https://example.com/robots.txt. The token AhrefsSiteAudit is matched case-insensitively.
robots.txt: block
# Block AhrefsSiteAudit
User-agent: AhrefsSiteAudit
Disallow: /robots.txt: allow
# Allow AhrefsSiteAudit
User-agent: AhrefsSiteAudit
Allow: /robots.txt: partial
# Keep AhrefsSiteAudit out of private areas only
User-agent: AhrefsSiteAudit
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to AhrefsSiteAudit
if ($http_user_agent ~* "(AhrefsSiteAudit)") {
return 403;
}Apache
# Apache .htaccess: return 403 to AhrefsSiteAudit
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AhrefsSiteAudit) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "AhrefsSiteAudit")How to verify real AhrefsSiteAudit traffic
Ahrefs publishes no IP list or reverse DNS pattern for AhrefsSiteAudit, so the user agent cannot be verified. Anyone can send the same string. Rate-limit unusual volume instead of trusting the name.
How to find AhrefsSiteAudit in your server logs
Search the web server access log for the token. The command below counts AhrefsSiteAudit requests per URL on Nginx or Apache.
grep -i "AhrefsSiteAudit" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20For a full report per AI bot, upload the log to the AI Log File Analyzer, a free tool by Justin Hà.
AhrefsSiteAudit FAQ
AhrefsSiteAudit uses the robots.txt token "AhrefsSiteAudit". Ahrefs does not publish a full user-agent string, so match the token inside the User-Agent header.
Yes. Ahrefs states that AhrefsSiteAudit follows robots.txt rules written for the "AhrefsSiteAudit" token.
Blocking an SEO tool crawler has no effect on Google or Bing rankings. No effect on AI answers. Blocking hides your pages and backlinks from that SEO tool, including from competitors who use it, and breaks your own audits in that tool.
Add two lines to the robots.txt file at the root of each domain and subdomain: "User-agent: AhrefsSiteAudit" followed by "Disallow: /". The token is case-insensitive. Changes can take up to 24 hours to be picked up.
Not reliably. Ahrefs publishes no IP list or reverse DNS pattern for AhrefsSiteAudit, so anyone can send the same user agent. Rate-limit unusual volume instead of trusting the name.
Search your server access log for "AhrefsSiteAudit", for example with: grep -i "AhrefsSiteAudit" access.log. The AI Log File Analyzer by Justin Hà counts visits per AI bot from an uploaded log file.
Sources: Ahrefs crawler documentation. Last reviewed by Justin Hà.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
