How to Block Some of the Bots • Hacker News Board

Re: How to Block Some of the Bots

#151

Post by watwut » Mon, Jul 27, 2026, 5:28 PM UTC

jolux wrote: post #101

razorbeamz wrote: post #93

Earlier quoted context omitted.

Do you have a source for those numbers?

ChatGPT has over a billion monthly active users.

I am monthly active user, absolutely do not experience web through ChatGPT alone and frankly, dont mind ChatGPT being cut off stuff. AI companies forcing themselves as a middle man is not a good thing.

Whoever using purely agentic worflow is not my concern. Just like people who read only physical book, learn only from podcasts or watch only netflix movies.

There is zero reason for the rest of us to worry about filling free data to companies that try to make themselves monopolies.

Re: How to Block Some of the Bots

#152

Post by 1vuio0pswjnm7 » Mon, Jul 27, 2026, 5:34 PM UTC

Need HTTP/2.0, User-Agent: and Sec-Fetch-Mode:

Response is brotli compressed, Accept-Encoding: is ignored

Overall, easy to make requests outside the browser, no need for Javascript

Distinguished name on the x509 certificate is obscene

Response headers are full of profanity, plus URL pointing to this video:

https://www.youtube.com/watch?v=wAk6BGgaSI4

Bananas is misspelled

Re: How to Block Some of the Bots

#153

Post by chlorion » Mon, Jul 27, 2026, 5:44 PM UTC

AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

My cgit instance gets hammered so hard that my CPU use is raised to 100% and the entire server grinds to a halt. I have to set a cgroups CPUQuota limit with systemd to make the server usable again.

Also, they hit so hard that nginx returns 502s and times out for real user half of the time, and when the site does load it takes a few seconds to load.

This is all on a high performance single core vultr server, not the cheapest vps but not the fastest, but should be plenty fast enough.

Re: How to Block Some of the Bots

#154

Post by exiguus » Mon, Jul 27, 2026, 10:01 PM UTC

In my view, if blocking is necessary, whitelisting is generally more effective than blacklisting. And if whitelisting isn't feasible, this might be a poor solution.

On the other hand, tools like Cloudflare and Anubis can create serous accessibility issues; some more than others.

I prefer rate limiting; it's elegant and avoids accessibility problems. For temporary issues, short-term IP bans can also work well.

Personally, I use fail2ban to parse HTTP logs and temporarily ban IPs for N hours if they request forbidden URLs (e.g., those blocked by robots.txt or paths like wp-login.php). Same if a IP trigger the rate limit much to often. And currently, I'm testing Anubis on my Git web UI.

Re: How to Block Some of the Bots

#155

Post by Bender » Mon, Jul 27, 2026, 10:25 PM UTC

exiguus wrote: post #154

In my view, if blocking is necessary, whitelisting is generally more effective than blacklisting. And if whitelisting isn't feasible, this might be a poor solution. On the other hand, tools like Cloudflare and Anubis can create serous accessibility issues; some more than others. I prefer rate limiting; it's elegant and avoids accessibility problems. For temporary issues, short-term IP bans can also work well. Persona…

I agree whitelisting is great. I have used that in B2B communications by setting up network to network VPN's. Anyone not on the VPN can not reach the servers. Employees can use their corporate VPN to also access the servers.

Re: How to Block Some of the Bots

#156

Post by ajsnigrutin » Tue, Jul 28, 2026, 12:24 AM UTC

BLKNSLVR wrote: post #82

inigyou wrote: post #59

Earlier quoted context omitted.

saying you shouldn't do it is also allowed

So long as a some half-decent explanation of reasoning is provided. Plenty of people would say that I shouldn't block residential proxies, but I would 100% block residential proxies if I could. I think most people responsible for blocking traffic would actively manage their rules if it was costing them something valuable. If they don't notice, then, well no harm no foul.

I gave my explanation, i sometimes have to download something to a server, and this means that I have to download it to my pc first and then scp it, sometimes over an extra server in the middle to get the file to where I want it.

Blocking vpns is the same kind of bad... with governments requiring you to show your ID or do a face scan just to jerk off, vpn usage will go up, and you'll be blocking all those people for no good reason.

Sure, you can block whatever you want, and i can and will hate you for that.

Re: How to Block Some of the Bots

#157

Post by iririririr » Wed, Jul 29, 2026, 5:08 PM UTC

inigyou wrote: post #62

ferngodfather wrote: post #16

Earlier quoted context omitted.

That's fair enough. I think it's a great article to be fair. We need more of this cheap and quick bot blocking. The fact the solution to unwanted traffic is often "use Cloudflare" is _not_ great for the internet, and nobody really actually likes deploying or managing ModSecurity. Its a nice middleground.

Cheap and quick bot-blocking is provided by go-away, less cheap and quick by Anubis.

both of these are easily bypassed by bots.

security theater at its finest. not to mention wasteful.

Re: How to Block Some of the Bots

#158

Post by ferngodfather » Fri, Jul 31, 2026, 5:23 AM UTC

iririririr wrote: post #157

inigyou wrote: post #62

Earlier quoted context omitted.

Cheap and quick bot-blocking is provided by go-away, less cheap and quick by Anubis.

both of these are easily bypassed by bots. security theater at its finest. not to mention wasteful.

It's not "security theatre", it's blocking the absolute worst, most painfully obvious bots that. These bots also happen to account for a good 85% of bot traffic, so it makes sense to block them.

Yes, of course, if a real threat actor is doing things properly they'll be using residential proxies and legitimate user agents, but we're talking here about blocking "some of the bots" - mainly the script kiddies.