451–460 of 516 posts
I think a lot of really smart people are letting themselves get taken for a ride by the web scraping thing. Unless the bot activity is legitimately hammering your site and causing issues (not saying this isn't happening in some cases), then this mostly amounts to an ideological game of capture the flag. The difference being that you'll never find their flag. The only thing you win by playing is lost time. The best wa…
My friend has a small public gitea instance, only use by him a a few friends. He's getting thousounds of requests an hour from bots. I'm sorry but even if it does not impact his service, at the very least it feels like harassment
This is a problem. There's a recent phishing campaign with sites hosted by Cloudflare and spam sent through either "noobtech.in" (103.173.40.0/24) or through "worldhost.group" (many, many networks). "noobtech.in" has no web site, can't accept abuse complaints (their email has spam filters), and they don't respond at all to email asking them for better communication methods. The phishing domains have "mail.(phishing d…
We solved a lot of our problems by blocking all Chinese ASNs. Admittedly, not the friendliest solution, but there were so many issues originating from Chinese clients that it was easier to just ban the entire country. It's not like we can capitalize on commerce in China anyway, so I think it's a fairly pragmatic approach.
Why stop there? Just block all non-US IPs! If it works for my health insurance company, essentially all streaming services (including not even being able to cancel service from abroad), and many banks, it’ll work for you as well. Surely bad actors wouldn’t use VPNs or botnets, and your customers never travel abroad?
They usually don't bother. Plus it's easier to take action against malicious traffic within your own country or general jurisdiction.
Earlier quoted context omitted.
Yes, and this makes ip banning have false positives. But ultimately it's worth it, you are responsible for your neighbours.
> [Y]ou are responsible for [how] your neighbours [use the Internet]. Nope. I'm very much not responsible for snooping on my neighbor's private communications. If anyone is responsible for doing any sort of abuse monitoring, it is the ISP chosen by my neighbor.
Wouldn't it be better, if there's an easy way, to just feed such bots shit data instead of blocking them. I know it's easier to block and saves compute and bandwidth, but perhaps feeding them shit data at scale would be a much better longer term solution.
if ($http_user_agent ~* "BadBot") { limit_rate 1k; default_type application/octet-stream; proxy_buffering off; alias /dev/zero; return 200; }
Earlier quoted context omitted.
No, it's really the same thing with just different (and more structured) prefix lengths. In IPv4 you usually block a single /32 address first, then a /24 block, etc. In IPv6 you start with a single /128 address, a single LAN is /64, an entire site is usually /56 (residential) or /48 (company), etc.
Note that for the sake of blocking internet clients, there's no point blocking a /128. Just start at /64. Blocking a /128 is basically useless because of SLAAC.
I would never have considered this, but someone on HN pointed out that web user agents work like this. Servers send ads and there is no way for them to enforce that browsers render the ads and hide the content or whatever. The user agent is supposed to act for the user. "Your business model is not my problem", etc. Well, my user agents work for me, not for you - the server guy who is complaining about this and that.…
Well done on pointing out exactly what everyone here is saying in the most arrogant way possible. Also, well done on linking to your own comment where people explain this to you. The problem is that there is no way to "block me if you don't want me". That's the entire issue. The methods these scrapers use mean it's nigh on impossible to block them.
I suspect we'll get integrity attestation or tokens before it becomes an unsurmountable problem to block bots.
Earlier quoted context omitted.
Doesn’t that tie up a socket on the server similarly to how a keepalive would on the bot user end?
I don't think so. The payload size of the bytes on the wire is small. This premise is all dependent on the .zip being crawled synchronously by the same thread/job making the request.
https://en.wikipedia.org/wiki/Zeno%27s_paradoxes#Dichotomy_p...
Earlier quoted context omitted.
It is funny how people immediately jump to conclusions, while I was merely pointing out an circular argument. People immediately think I am jumping to "aid one side". Shows much more about them than me, actually.
Nobody is jumping to any conclusions except you with this very comment. ;)