Twitterbot user agent
Operated by X
Twitterbot is the fetcher X uses to build the card preview when a link is posted on X.
The Twitterbot bot at a glance
- Twitterbot is operated by X and reads page titles, descriptions and images to build link previews.
- The robots.txt token for Twitterbot is Twitterbot.
- Blocking a social preview bot has no effect on search rankings.
- Twitterbot robots.txt support: Yes.
- Recommendation for Generative Engine Optimization (GEO): Allow.
What is Twitterbot?
Twitterbot reads a page’s Twitter Card and Open Graph tags to show its title, description and image in posts on X.
X no longer publishes its Cards crawler page, so this profile relies on the open crawler-user-agents list.
Should you block Twitterbot?
Allow Twitterbot unless it causes a measurable load problem.
| Effect on search rankings | Blocking a social preview bot has no effect on search rankings. |
|---|---|
| Effect on AI visibility | No direct effect on AI answers, but shared links lose their title, image and description, which usually lowers clicks from social and chat apps. |
| Follows robots.txt | Yes |
| Recommendation | Allow |
How to block or allow Twitterbot
Put the robots.txt rule in the file at the root of every domain and subdomain, for example https://example.com/robots.txt. The token Twitterbot is matched case-insensitively.
robots.txt: block
# Block Twitterbot
User-agent: Twitterbot
Disallow: /robots.txt: allow
# Allow Twitterbot
User-agent: Twitterbot
Allow: /robots.txt: partial
# Keep Twitterbot out of private areas only
User-agent: Twitterbot
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to Twitterbot
if ($http_user_agent ~* "(Twitterbot)") {
return 403;
}Apache
# Apache .htaccess: return 403 to Twitterbot
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Twitterbot) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "Twitterbot")How to verify real Twitterbot traffic
X publishes no IP list or reverse DNS pattern for Twitterbot, so the user agent cannot be verified. Anyone can send the same string. Rate-limit unusual volume instead of trusting the name.
How to find Twitterbot in your server logs
Search the web server access log for the token. The command below counts Twitterbot requests per URL on Nginx or Apache.
grep -i "Twitterbot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20For a full report per AI bot, upload the log to the AI Log File Analyzer, a free tool by Justin Hà.
Twitterbot user agent string
The full user agent X publishes for Twitterbot. Version numbers such as Chrome/W.X.Y.Z change over time, so match the token, not the whole string.
Twitterbot/1.0Twitterbot FAQ
Twitterbot identifies itself with the robots.txt token "Twitterbot". A typical request header reads: Twitterbot/1.0
Yes. X states that Twitterbot follows robots.txt rules written for the "Twitterbot" token.
Blocking a social preview bot has no effect on search rankings. No direct effect on AI answers, but shared links lose their title, image and description, which usually lowers clicks from social and chat apps.
Add two lines to the robots.txt file at the root of each domain and subdomain: "User-agent: Twitterbot" followed by "Disallow: /". The token is case-insensitive. Changes can take up to 24 hours to be picked up.
Not reliably. X publishes no IP list or reverse DNS pattern for Twitterbot, so anyone can send the same user agent. Rate-limit unusual volume instead of trusting the name.
Search your server access log for "Twitterbot", for example with: grep -i "Twitterbot" access.log. The AI Log File Analyzer by Justin Hà counts visits per AI bot from an uploaded log file.
Sources: Public bot registry. Last reviewed by Justin Hà.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
