BotBouncerBotBouncer / AI bots / Meta-ExternalFetcher
Meta-ExternalFetcher
Meta, a fetcher that runs when a user asks
- Operator
- Meta
- Type
- Fetches on a user's request
- Sent as a user agent
- Not confirmed
- Follows robots.txt
- Partly
- Details
- Not yet checked against the operator's own documentation
What it does
Fetches links on a user's request. Meta says it may bypass robots.txt.
We have not yet checked this page against Meta's own documentation. Check it before you rely on it.
Block it in robots.txt
User-agent: Meta-ExternalFetcher Disallow: /
To allow it again, remove those lines or use Allow: /. Rules for other bots are not affected.
Block it on your server
robots.txt is a request. To enforce the block, filter the user agent on your server.
Nginx
map $http_user_agent $blocked_bot {
default 0;
~*(Meta-ExternalFetcher) 1;
}
# inside server { }
if ($blocked_bot) { return 403; }Apache (.htaccess)
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Meta-ExternalFetcher [NC]
RewriteRule ^ - [F,L]Cloudflare custom rule (action: Block)
(lower(http.user_agent) contains "meta-externalfetcher")
Source
No official documentation link has been confirmed yet.