Bots by company
Ai2 bots and crawlers
Ai2 (Allen Institute for AI) is a non-profit research institute that trains open language models.
Ai2 bots at a glance
- Ai2 runs 3 bots listed in this directory.
- They belong to: AI Training Crawlers, AI Assistant Fetchers.
- Each Ai2 bot has its own robots.txt token, so each purpose can be allowed or blocked separately.
- Website: allenai.org
Every Ai2 bot
| Bot | Group | Purpose | robots.txt | For GEO |
|---|---|---|---|---|
| AI2Bot | AI Training Crawlers | AI trainingOfficial docs | Yes | Your choice |
| AI2Bot-DeepResearchEval | AI Assistant Fetchers | AI assistantReported | Not documented | Your choice |
| Ai2Bot-Dolma | AI Training Crawlers | AI trainingReported | Yes | Your choice |
Block or allow all Ai2 bots
The rules below name every Ai2 token in this directory. To block training but keep AI search, delete the search and assistant tokens from the block rule.
robots.txt: block
# Block all Ai2 bots
User-agent: AI2Bot
User-agent: AI2Bot-DeepResearchEval
User-agent: Ai2Bot-Dolma
Disallow: /robots.txt: allow
# Allow all Ai2 bots
User-agent: AI2Bot
User-agent: AI2Bot-DeepResearchEval
User-agent: Ai2Bot-Dolma
Allow: /robots.txt: partial
# Keep all Ai2 bots out of private areas only
User-agent: AI2Bot
User-agent: AI2Bot-DeepResearchEval
User-agent: Ai2Bot-Dolma
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to all Ai2 bots
if ($http_user_agent ~* "(AI2Bot|AI2Bot\-DeepResearchEval|Ai2Bot\-Dolma)") {
return 403;
}Apache
# Apache .htaccess: return 403 to all Ai2 bots
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (AI2Bot|AI2Bot\-DeepResearchEval|Ai2Bot\-Dolma) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "AI2Bot") or (http.user_agent contains "AI2Bot-DeepResearchEval") or (http.user_agent contains "Ai2Bot-Dolma")See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
