Archives and Research Crawlers
ia_archiver user agent
Operated by Alexa Internet
ia_archiver is the legacy crawler of Alexa Internet, which also contributed captures to the Internet Archive.
The ia_archiver bot at a glance
- ia_archiver is operated by Alexa Internet and fetches pages to power a specific product feature.
- The robots.txt token for ia_archiver is ia_archiver.
- Blocking an archive crawler has no effect on search rankings.
- ia_archiver robots.txt support: Not documented.
- Recommendation for Generative Engine Optimization (GEO): Allow.
What is ia_archiver?
Alexa Internet was retired in 2022, but many robots.txt files still name ia_archiver.
Keeping a rule for it is harmless; archive.org_bot is the active Internet Archive crawler.
Should you block ia_archiver?
Allow ia_archiver unless it causes a measurable load problem.
| Effect on search rankings | Blocking an archive crawler has no effect on search rankings. |
|---|---|
| Effect on AI visibility | Archived copies can be reused by researchers and AI datasets; blocking stops new snapshots but does not remove existing ones. |
| Follows robots.txt | Not documented |
| Recommendation | Allow |
How to block or allow ia_archiver
Put the robots.txt rule in the file at the root of every domain and subdomain, for example https://example.com/robots.txt. The token ia_archiver is matched case-insensitively. robots.txt alone may not stop ia_archiver, so add the Nginx, Apache or Cloudflare rule as well.
robots.txt: block
# Block ia_archiver
User-agent: ia_archiver
Disallow: /robots.txt: allow
# Allow ia_archiver
User-agent: ia_archiver
Allow: /robots.txt: partial
# Keep ia_archiver out of private areas only
User-agent: ia_archiver
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to ia_archiver
if ($http_user_agent ~* "(ia_archiver)") {
return 403;
}Apache
# Apache .htaccess: return 403 to ia_archiver
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (ia_archiver) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "ia_archiver")How to verify real ia_archiver traffic
Alexa Internet publishes no IP list or reverse DNS pattern for ia_archiver, so the user agent cannot be verified. Anyone can send the same string. Rate-limit unusual volume instead of trusting the name.
How to find ia_archiver in your server logs
Search the web server access log for the token. The command below counts ia_archiver requests per URL on Nginx or Apache.
grep -i "ia_archiver" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20For a full report per AI bot, upload the log to the AI Log File Analyzer, a free tool by Justin Hà.
ia_archiver user agent string
The full user agent Alexa Internet publishes for ia_archiver. Version numbers such as Chrome/W.X.Y.Z change over time, so match the token, not the whole string.
ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])ia_archiver FAQ
ia_archiver identifies itself with the robots.txt token "ia_archiver". A typical request header reads: ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])
Alexa Internet has not documented how ia_archiver treats robots.txt. Add a robots.txt rule and back it with a server rule if the traffic matters to you.
Blocking an archive crawler has no effect on search rankings. Archived copies can be reused by researchers and AI datasets; blocking stops new snapshots but does not remove existing ones.
Add two lines to the robots.txt file at the root of each domain and subdomain: "User-agent: ia_archiver" followed by "Disallow: /". The token is case-insensitive. Changes can take up to 24 hours to be picked up.
Not reliably. Alexa Internet publishes no IP list or reverse DNS pattern for ia_archiver, so anyone can send the same user agent. Rate-limit unusual volume instead of trusting the name.
Search your server access log for "ia_archiver", for example with: grep -i "ia_archiver" access.log. The AI Log File Analyzer by Justin Hà counts visits per AI bot from an uploaded log file.
Sources: Public bot registry. Last reviewed by Justin Hà.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
