Using Nginx to block Meta, Twitter and ChatGPT access to your sites

If playing around with blocking bots here is one that will block some legit and friendly bots and some bad ones too. This will do nothing for headless chrome acting as a bot. Only use this on silly hobby sites, do not use in production even though most load balancers can do this and much more.

In the main site config redirect anyone not using HTTP/2.0. GoogleBot still doesnt use HTTP/2.0 so this will block Google. Bing is OK though. One could instead use variables to make this multi-condition and make exceptions for their CIDR blocks. Point "auth." DNS record to the same IP and ensure you have a cert for it or a wildcard cert.

    # in main TLS site config:
    # replace apex with your domain and tld with its tld.
    if ($server_protocol != HTTP/2.0) { return 302 https://auth.apex.tld$request_uri; }

Then in your "auth" domain use the same config as the main site minus the redirect but then add basic authentication. Anyone not using HTTP/2.0 can still access the site if they know the right username/password. If you get a lot of bots then have an init script copy the password file into /dev/shm and reference it from there in NGinx to avoid the disk reads.

    # then in the auth.apex.tld config.
    # optionally give a hint replacing i_heart_bots with name_blah_pass_blah
    auth_delay 2s;
    location / {
    auth_basic "i_heart_bots"; auth_basic_user_file /etc/nginx/.pw;
    }

This will block some API command line tools, most bots good or bad, some scanning tools. Some bots will give up prior to 2 seconds so you will get a status 499 instead of 401 in the access logs. Only do this on silly hobby sites. Do not use in production. Only people wearing a T-Shirt like this one [1] may do this in production.

One may be surprised to find that most bots use old libraries that are not HTTP/2.0 enabled. When they catch up we can replace this logic using HTTP/3.0 and UDP. Beyond that we can force people to win a game of tic-tac-toe or Doom over Javascript.

[1] - https://www.amazon.com/Dont-Always-Test-Production-Shirt/dp/...

Why would they be? No contract was signed, no DRM was being broken, they’re simply accessing the website you’re publicly hosting.

There are a lot of dumb bots.

And you think Meta or Twitter's bots would be dumb ?

If they wanted to scrape your site, nobody can prevent them.

robots.txt isn’t honoured only by bad bots and scrapers, so I agree with you that this nginx configuration is pretty useless and doesn’t even solve the problem on the other side!