Bytespider user agent
Operated by ByteDance
Bytespider is ByteDance's crawler, widely reported as collecting web data to train ByteDance language models.
The Bytespider bot at a glance
- Bytespider is operated by ByteDance and collects public web pages that may be used to train AI models.
- The robots.txt token for Bytespider is Bytespider.
- Blocking an AI training crawler has no effect on Google or Bing rankings.
- Bytespider robots.txt support: No.
- Recommendation for Generative Engine Optimization (GEO): Consider blocking.
What is Bytespider?
Bytespider is one of the highest-volume AI crawlers seen in server logs, and public reports describe it crawling sites that disallow it.
Sites that want it gone usually add a firewall or server rule in addition to robots.txt.
Should you block Bytespider?
Consider blocking Bytespider: it brings little visibility back and its robots.txt behaviour is weak or undocumented.
| Effect on search rankings | Blocking an AI training crawler has no effect on Google or Bing rankings. |
|---|---|
| Effect on AI visibility | Blocking keeps your future pages out of that company's training data, so its models may know less about your brand over time. It does not remove you from that company's AI search, which uses a separate bot. |
| Follows robots.txt | No |
| Recommendation | Consider blocking |
How to block or allow Bytespider
Put the robots.txt rule in the file at the root of every domain and subdomain, for example https://example.com/robots.txt. The token Bytespider is matched case-insensitively. robots.txt alone may not stop Bytespider, so add the Nginx, Apache or Cloudflare rule as well.
robots.txt: block
# Block Bytespider
User-agent: Bytespider
Disallow: /robots.txt: allow
# Allow Bytespider
User-agent: Bytespider
Allow: /robots.txt: partial
# Keep Bytespider out of private areas only
User-agent: Bytespider
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to Bytespider
if ($http_user_agent ~* "(Bytespider)") {
return 403;
}Apache
# Apache .htaccess: return 403 to Bytespider
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Bytespider) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "Bytespider")How to verify real Bytespider traffic
ByteDance publishes no IP list or reverse DNS pattern for Bytespider, so the user agent cannot be verified. Anyone can send the same string. Rate-limit unusual volume instead of trusting the name.
How to find Bytespider in your server logs
Search the web server access log for the token. The command below counts Bytespider requests per URL on Nginx or Apache.
grep -i "Bytespider" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20For a full report per AI bot, upload the log to the AI Log File Analyzer, a free tool by Justin Hà.
Bytespider FAQ
Bytespider uses the robots.txt token "Bytespider". ByteDance does not publish a full user-agent string, so match the token inside the User-Agent header.
No reliable evidence says it does. Reports show Bytespider crawling sites that disallow it, so pair robots.txt with a firewall or server rule.
Blocking an AI training crawler has no effect on Google or Bing rankings. Blocking keeps your future pages out of that company's training data, so its models may know less about your brand over time. It does not remove you from that company's AI search, which uses a separate bot.
Add two lines to the robots.txt file at the root of each domain and subdomain: "User-agent: Bytespider" followed by "Disallow: /". The token is case-insensitive. Changes can take up to 24 hours to be picked up.
Not reliably. ByteDance publishes no IP list or reverse DNS pattern for Bytespider, so anyone can send the same user agent. Rate-limit unusual volume instead of trusting the name.
Search your server access log for "Bytespider", for example with: grep -i "Bytespider" access.log. The AI Log File Analyzer by Justin Hà counts visits per AI bot from an uploaded log file.
Sources: Public bot registry. Last reviewed by Justin Hà.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
