Bot Directory
Archives and Research Crawlers
Archive and research crawlers copy public pages into long-term collections such as the Internet Archive's Wayback Machine, national web archives and academic datasets. Archived copies are often used as historical evidence and can end up in AI training data.
Archives and Research Crawlers at a glance
- This group lists 4 bots.
- Blocking an archive crawler has no effect on search rankings.
- Archived copies can be reused by researchers and AI datasets; blocking stops new snapshots but does not remove existing ones.
All Archives and Research Crawlers
| Bot | Company | Purpose | robots.txt | For GEO |
|---|---|---|---|---|
| archive.org_bot | Internet Archive | Product featureReported | Not documented | Allow |
| Arquivo-web-crawler | Arquivo.pt | Product featureReported | Not documented | Allow |
| ia_archiver | Alexa Internet | Product featureReported | Not documented | Allow |
| TurnitinBot | Turnitin | Product featureReported | Not documented | Your choice |
Block or allow every bot in Archives and Research Crawlers at once
The rules below name all 4 robots.txt tokens in this group. Remove the lines for any bot you want to treat differently.
robots.txt: block
# Block all Archives and Research Crawlers
User-agent: archive.org_bot
User-agent: Arquivo-web-crawler
User-agent: ia_archiver
User-agent: TurnitinBot
Disallow: /robots.txt: allow
# Allow all Archives and Research Crawlers
User-agent: archive.org_bot
User-agent: Arquivo-web-crawler
User-agent: ia_archiver
User-agent: TurnitinBot
Allow: /robots.txt: partial
# Keep all Archives and Research Crawlers out of private areas only
User-agent: archive.org_bot
User-agent: Arquivo-web-crawler
User-agent: ia_archiver
User-agent: TurnitinBot
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to all Archives and Research Crawlers
if ($http_user_agent ~* "(archive\.org_bot|Arquivo\-web\-crawler|ia_archiver|TurnitinBot)") {
return 403;
}Apache
# Apache .htaccess: return 403 to all Archives and Research Crawlers
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (archive\.org_bot|Arquivo\-web\-crawler|ia_archiver|TurnitinBot) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "archive.org_bot") or (http.user_agent contains "Arquivo-web-crawler") or (http.user_agent contains "ia_archiver") or (http.user_agent contains "TurnitinBot")Other bot groups
AI Training Crawlers
44 bots
Collect public pages that may be used to train AI models. Blocking them has no effect on Google or Bing rankings.
AI Search Crawlers
21 bots
Index pages so AI search products can show and cite them. Blocking them removes you from those answers.
AI Assistant Fetchers
30 bots
Open a page when a user asks an AI assistant about it. They act for one person, not for an index.
AI Agents
14 bots
Use a browser to complete tasks for users, such as comparing, booking or buying.
Search Engine Crawlers
60 bots
Index pages for classic search engines. Blocking them removes you from search results.
Ads and Platform Fetchers
38 bots
Serve one platform feature: ad checks, feeds, verification or safety scans.
SEO Tool Crawlers
29 bots
Crawl the web for SEO platforms: backlink indexes, rank data and site audits. Blocking them hides your site from those tools.
Social and Messaging Previews
22 bots
Build the preview card when your link is shared in a social or messaging app. Blocking them breaks those cards.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
