Meta-ExternalFetcher user agent
Operated by Meta
Meta-ExternalFetcher is the user agent Meta AI products use to fetch a specific page a user asked about.
The Meta-ExternalFetcher bot at a glance
- Meta-ExternalFetcher is operated by Meta and fetches a page when a person asks an AI assistant about it or shares its link.
- The robots.txt token for Meta-ExternalFetcher is meta-externalfetcher.
- Blocking an AI assistant fetcher has no effect on search rankings.
- Meta-ExternalFetcher robots.txt support: Not for user-requested fetches.
- Recommendation for Generative Engine Optimization (GEO): Allow.
What is Meta-ExternalFetcher?
Meta-ExternalFetcher fetches individual links at a user’s request, and Meta says it also supports agentic AI features that help AI complete tasks on websites for users.
Because a user requested the visit, Meta says it may bypass robots.txt rules.
Should you block Meta-ExternalFetcher?
Allow Meta-ExternalFetcher unless it causes a measurable load problem.
| Effect on search rankings | Blocking an AI assistant fetcher has no effect on search rankings. |
|---|---|
| Effect on AI visibility | Blocking stops the assistant from reading your page when a user asks about it or pastes your link, so its answer falls back to other sources. |
| Follows robots.txt | Not for user-requested fetches |
| Recommendation | Allow |
How to block or allow Meta-ExternalFetcher
Put the robots.txt rule in the file at the root of every domain and subdomain, for example https://example.com/robots.txt. The token meta-externalfetcher is matched case-insensitively. robots.txt alone may not stop Meta-ExternalFetcher, so add the Nginx, Apache or Cloudflare rule as well.
robots.txt: block
# Block Meta-ExternalFetcher
User-agent: meta-externalfetcher
Disallow: /robots.txt: allow
# Allow Meta-ExternalFetcher
User-agent: meta-externalfetcher
Allow: /robots.txt: partial
# Keep Meta-ExternalFetcher out of private areas only
User-agent: meta-externalfetcher
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Allow: /Nginx
# Nginx: return 403 to Meta-ExternalFetcher
if ($http_user_agent ~* "(meta\-externalfetcher)") {
return 403;
}Apache
# Apache .htaccess: return 403 to Meta-ExternalFetcher
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (meta\-externalfetcher) [NC]
RewriteRule .* - [F,L]Cloudflare WAF
# Cloudflare > Security > WAF > Custom rule, action: Block
(http.user_agent contains "meta-externalfetcher")How to verify real Meta-ExternalFetcher traffic
Meta publishes no IP list or reverse DNS pattern for Meta-ExternalFetcher, so the user agent cannot be verified. Anyone can send the same string. Rate-limit unusual volume instead of trusting the name.
How to find Meta-ExternalFetcher in your server logs
Search the web server access log for the token. The command below counts Meta-ExternalFetcher requests per URL on Nginx or Apache.
grep -i "meta-externalfetcher" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20For a full report per AI bot, upload the log to the AI Log File Analyzer, a free tool by Justin Hà.
Meta-ExternalFetcher user agent string
The full user agent Meta publishes for Meta-ExternalFetcher. Version numbers such as Chrome/W.X.Y.Z change over time, so match the token, not the whole string.
meta-externalfetcher/1.1Meta-ExternalFetcher FAQ
Meta-ExternalFetcher identifies itself with the robots.txt token "meta-externalfetcher". A typical request header reads: meta-externalfetcher/1.1
Partly. Meta-ExternalFetcher fetches pages because a user asked for them, and Meta says robots.txt rules may not apply to those user-requested fetches. Use a firewall rule if you must stop it.
Blocking an AI assistant fetcher has no effect on search rankings. Blocking stops the assistant from reading your page when a user asks about it or pastes your link, so its answer falls back to other sources.
Add two lines to the robots.txt file at the root of each domain and subdomain: "User-agent: meta-externalfetcher" followed by "Disallow: /". The token is case-insensitive. Changes can take up to 24 hours to be picked up.
Not reliably. Meta publishes no IP list or reverse DNS pattern for Meta-ExternalFetcher, so anyone can send the same user agent. Rate-limit unusual volume instead of trusting the name.
Search your server access log for "meta-externalfetcher", for example with: grep -i "meta-externalfetcher" access.log. The AI Log File Analyzer by Justin Hà counts visits per AI bot from an uploaded log file.
Sources: Meta crawler documentation. Last reviewed by Justin Hà.
See which of these bots actually reach your website
The AI Log File Analyzer by Justin Hà counts visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in your server log, for free. The GEO Technical Package sets up robots.txt, llms.txt and crawler access for AI search.
