Our robots

fail2ban

Add our address to ignoreip in jail.local, under [DEFAULT] to cover every jail. A jail's own ignoreip replaces the default rather than adding to it, so repeat the address there if you set one.

# /etc/fail2ban/jail.local
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 34.53.138.166

Reload, and lift a ban already in place:

sudo fail2ban-client reload
sudo fail2ban-client unban 34.53.138.166

Prefer ignoreip to an ignoreregex on our User-Agent: most jails (apache-botsearch among them) read log lines that do not contain the User-Agent, so the regex never matches.

CrowdSec

On CrowdSec 1.6.8 or later, add our address to an allowlist. It applies at once:

sudo cscli allowlists create globetrotters -d "Globetrotters robots"
sudo cscli allowlists add globetrotters 34.53.138.166 -d "https://globetrotters.ai/bot"

On earlier versions, use a parser whitelist:

# /etc/crowdsec/parsers/s02-enrich/globetrotters-whitelist.yaml
name: globetrotters/whitelist
description: "Globetrotters robots - https://globetrotters.ai/bot"
whitelist:
  reason: "Globetrotters robots"
  ip:
    - "34.53.138.166"

Then reload, and remove a decision already in place:

sudo systemctl reload crowdsec
sudo cscli decisions delete --ip 34.53.138.166

Apache 2.4

With ModSecurity, turn the rule engine off for our address. Place the rule before your rule set is loaded (with the OWASP CRS, in REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf), and pick an id that is free on your server:

SecRule REMOTE_ADDR "@ipMatch 34.53.138.166" \
    "id:1000001,phase:1,pass,nolog,ctl:ruleEngine=Off"

With mod_evasive, add it to the module's configuration:

DOSWhitelist 34.53.138.166

For a location restricted by IP, add it next to your existing rules:

<Location "/private">
    <RequireAny>
        Require ip 34.53.138.166
        # ...your existing Require lines
    </RequireAny>
</Location>

nginx

In a location restricted by IP, allow our address before the deny:

location /private/ {
    allow 34.53.138.166;
    # ...your existing allow lines
    deny all;
}

Cloudflare

Create a WAF custom rule (Security → WAF → Custom rules) with this expression and the action Skip, skipping the remaining custom rules, rate limiting rules, managed rules and Super Bot Fight Mode:

(ip.src eq 34.53.138.166)

Bot Fight Mode on the Free plan cannot be skipped by a custom rule. Add an IP Access rule for our address with the action Allow instead (Security → WAF → Tools).