Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Block Some of the Bots

nochan.net

Post reply on HN
Print view View original

131–140 of 158 posts

Re: How to Block Some of the Bots

#131

Post by inigyou »

thayne wrote: post #117

You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP. Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers. > One class of bot…

Author has indicated he's perfectly fine with blocking all sorts of human users. I wouldn't take any of his advice.

Re: How to Block Some of the Bots

#132

Post by inigyou »

Ndymium wrote: post #110
AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

In my case it's my Forgejo instance. My blog is static files with a limited amount of pages to crawl and indeed it's no issue at all. Forgejo is a dynamic service with an infinite amount of pages to discover and it runs git in the background to generate (some of) the pages. Tons of bots can easily overwhelm my small server. I could hide the repositories, but they're open on purpose (it's open source after all). I hav…

That seems fine. You are using a proportionate response to an actual problem.

Re: How to Block Some of the Bots

#133

Post by inigyou »

nubinetwork wrote: post #114
AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

At home, I can't afford a 40gbit connection and a server to match... getting DDoSed by a handful of malicious google cloud VPSes running nmap and various web vulnerability scans can easily impact performance on low end hardware. Yes, in the grand scheme of things, waiting a few additional seconds to check my local imap server (because my DMZ gets bogged down) isn't a huge deal, but that doesn't mean I have to like it…

Are you getting 40gbits of bot traffic?

Re: How to Block Some of the Bots

#134

Post by inigyou »

mkj wrote: post #105
AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things. Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine! But getting 2.7 million requests/day from…

> 1TB/month

Switch VPS providers, you can easily find higher numbers or "unlimited".

Re: How to Block Some of the Bots

#135

Post by inigyou »

Barbing wrote: post #108
mkj wrote: post #105

Earlier quoted context omitted.

It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things. Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine! But getting 2.7 million requests/day from…

30/s is preposterous. Are these loser companies who don’t want to cache on their end or are they getting unique info per request or could it really be that many different origins or…?

None of the above. They're just really retarded bots. Nobody knows why they exist or what they gain from really stupid scraping.

Re: How to Block Some of the Bots

#136

Post by inigyou »

Barbing wrote: post #122
yjftsjthsd-h wrote: post #118

Earlier quoted context omitted.

> Are these loser companies who don’t want to cache on their end Google couldn't be bothered to cache; I would guess this before anything else. https://drewdevault.com/blog/Google-has-been-DDoSing-sourceh...

> each IP address still clones the same repositories 8-10 times per hour. Dumb! When was this fixed and don’t say never :)

That just sounds like a CI system. Surely almost every one of us is responsible for at least one CI process that doesn't cache dependencies.

Re: How to Block Some of the Bots

#137

Post by inigyou »

Bender wrote: post #98
AussieWog93 wrote: post #83

Earlier quoted context omitted.

Not pointing and laughing, just wondering why (since I see so many posts about this). Sounds like it's just for the thrill of the game, which is completely valid.

Thrill of the game is a great way to put it.

I wish it was framed as "silly ways to block bots" then so that people wouldn't take it as a recommendation

Re: How to Block Some of the Bots

#138

Post by inigyou »

c2h5oh wrote: post #123
AussieWog93 wrote: post #66

I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots? WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that. Are you using lambdas or something like that to serve a blog? Is it basically just OCD? Vulnerability…

My dedicated server comes with 2TB/month data included. Humans use up about 2% of that. If I didn't block bots I'd run out mid-month.

On a dedicated server? It must be somewhere like Korea, where bandwidth is extremely expensive? Usually, dedicated servers are already expensive enough that they throw in unlimited bandwidth for free. And when not, its usually more like 200TB/m

Re: How to Block Some of the Bots

#139

Post by inigyou »

Gigachad wrote: post #104
userbinator wrote: post #37

You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago. Block on traffic volume and request frequency if that's causing a problem. (And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with…

The user agent string is harmful in general. Any new browser should just copy the chrome user agent.

That's why Google introduced this secure key thingy so that you'd have to claim to be Chrome in a way that Google hopes is either legally binding by itself or a copyright infringement of the keying algorithm.

I don't remember what it was called and I don't use Chrome to test it. It sends a header with a long hex string that supposedly proves it's real Chrome.

Re: How to Block Some of the Bots

#140

Post by inigyou »

63stack wrote: post #130
inigyou wrote: post #67

Earlier quoted context omitted.

Not reliably enough. Look for parser differentials.

Please enlighten us

Differences in parsing between two systems can allow data to sneak through. For example you might put a space before the header name. The reverse proxy might treat it as a header name starting with a space, so not matching the one it wants to remove, and not removed, while the origin server trims the space.

This is a ubiquitous security concern, like IDOR and memory safety. Every time you have a separate validation step for anything, you have the possibility of a parser differential vulnerability.

If you don't know it yet, look up "HTTP request smuggling" - pretty interesting.

Post reply on HN