Bots by company
Screaming Frog bots and crawlers
Screaming Frog makes the SEO Spider desktop crawler used for technical site audits.
Screaming Frog bots at a glance
- Screaming Frog runs 1 bot listed in this directory.
- They belong to: SEO Tool Crawlers.
- Each Screaming Frog bot has its own robots.txt token, so each purpose can be allowed or blocked separately.
- Website: www.screamingfrog.co.uk
Every Screaming Frog bot
| Bot | Group | Purpose | robots.txt | For GEO |
|---|---|---|---|---|
| Screaming Frog SEO Spider | SEO Tool Crawlers | Product featureOfficial docs | Yes | Your choice |
Block or allow all Screaming Frog bots
The rules below name every Screaming Frog token in this directory. To block training but keep AI search, delete the search and assistant tokens from the block rule.
robots.txt: block
# Block all Screaming Frog bots
User-agent: Screaming Frog SEO Spider
Disallow: /robots.txt: allow
# Allow all Screaming Frog bots
User-agent: Screaming Frog SEO Spider
Allow: /robots.txt: partial
# Keep all Screaming Frog bots out of private areas only
User-agent: Screaming Frog SEO Spider
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to all Screaming Frog bots
if ($http_user_agent ~* "(Screaming Frog SEO Spider)") {
return 403;
}Apache
# Apache .htaccess: return 403 to all Screaming Frog bots
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Screaming Frog SEO Spider) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "Screaming Frog SEO Spider")See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
