Bot Directory

AI bot & crawler directory

This directory lists 262 AI crawlers and search engine bots from 118 companies. Each profile gives the user agent, the robots.txt token, what blocking it does to SEO and AI visibility, and copy-ready rules to block or allow it.

The directory at a glance

143 of 262 bots have official documentation from their operator; the rest are reported by public sources.
Blocking an AI training crawler such as GPTBot or ClaudeBot has no effect on Google or Bing rankings.
Blocking an AI search crawler such as OAI-SearchBot or PerplexityBot removes your pages from that AI product's cited answers.
Blocking a search engine crawler such as Googlebot removes your pages from that search engine and the AI features built on it.

AI Training Crawlers

Collect public pages that may be used to train AI models. Blocking them has no effect on Google or Bing rankings.

All 44 AI Training Crawlers with a combined block list

AI Search Crawlers

Index pages so AI search products can show and cite them. Blocking them removes you from those answers.

All 21 AI Search Crawlers with a combined block list

AI Assistant Fetchers

Open a page when a user asks an AI assistant about it. They act for one person, not for an index.

All 30 AI Assistant Fetchers with a combined block list

AI Agents

Use a browser to complete tasks for users, such as comparing, booking or buying.

All 14 AI Agents with a combined block list

Search Engine Crawlers

Index pages for classic search engines. Blocking them removes you from search results.

All 60 Search Engine Crawlers with a combined block list

Ads and Platform Fetchers

Serve one platform feature: ad checks, feeds, verification or safety scans.

All 38 Ads and Platform Fetchers with a combined block list

SEO Tool Crawlers

Crawl the web for SEO platforms: backlink indexes, rank data and site audits. Blocking them hides your site from those tools.

All 29 SEO Tool Crawlers with a combined block list

Archives and Research Crawlers

Save public pages for web archives, libraries and research. Blocking them stops new snapshots of your site.

archive.org_bot Internet Archive Product featureReported Allow Arquivo-web-crawler Arquivo.pt Product featureReported Allow ia_archiver Alexa Internet Product featureReported Allow TurnitinBot Turnitin Product featureReported Your choice

All 4 Archives and Research Crawlers with a combined block list

Build the preview card when your link is shared in a social or messaging app. Blocking them breaks those cards.

All 22 Social and Messaging Previews with a combined block list

Bots by company

Each company page lists every bot that company runs and one robots.txt block that covers all of them.

How this directory labels each bot

Official docs

The operating company names the bot on its own documentation page. The profile links to that exact page, and its user agent, IP list and robots.txt behaviour come from it.

Reported

No operator page names the bot, but an open source list does: the ai.robots.txt registry or the crawler-user-agents list, both under the MIT License. Treat these profiles as best available information.

For GEO

Allow keeps AI search and assistants able to cite you. Your choice is a content-protection decision. Consider blocking marks bots with weak robots.txt behaviour.

AI bot FAQ

What is the difference between an AI crawler and a search engine crawler?+

A search engine crawler such as Googlebot indexes pages for classic search results, and blocking it removes you from that engine. An AI training crawler such as GPTBot collects pages for model training, and blocking it has no effect on Google or Bing rankings.

Which AI bots should I allow for GEO?+

Allow the AI search and assistant bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) if you want ChatGPT, Claude and Perplexity to cite your pages. Blocking training crawlers is a separate choice that does not remove you from AI search.

Does blocking GPTBot remove my site from ChatGPT?+

No. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT search, and each follows its own robots.txt rule. Blocking only GPTBot keeps your pages out of future training while ChatGPT search can still find and cite them.

What does "Official docs" mean in this directory?+

Official docs means the operating company names the bot on its own documentation page, and the profile links to that page. Reported means only a public source such as the ai.robots.txt registry names it, so its purpose and robots.txt behaviour are not confirmed by the operator.

Is robots.txt enough to stop AI bots?+

robots.txt is a request, not a lock. Well-behaved bots from OpenAI, Anthropic, Google and Perplexity follow it, but bots marked "No" or "Not documented" here may not, so back robots.txt with a firewall rule, such as a Cloudflare WAF custom rule.

See which of these bots actually reach your website

The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.

Analyze your AI bot traffic free See the GEO Technical Package