101–110 of 516 posts
Earlier quoted context omitted.
And that's perfectly fine. Nothing is completely bulletproof anyway. If you manage to get rid of 90% of the problem then that's a good thing.
And if your competitor manages to do so without annoying the part of their customer base that occasionally leaves the country, everybody wins!
Earlier quoted context omitted.
The origin of the term 'black list' had absolutely nothing to do with the melanin content of anyone . In fact, when that term was coined, it had nothing to do with the melanin content of anyone . It was a list of the enemies of Charles II. That's why I posted that. I'd also point out that in my lifetime , folks with darker skin called themselves black and proudly so. As Mr. Brown[0][1] will unambiguously tell you. Re…
[flagged]
My cat has a black tail.
The top of my desk is black.
I have several pairs of black shoes.
Every single computer in my possession has a black case.
My phone and its case are both black.
Black Power![0][1][2]
I will put you on my personal blacklist.
Which I'm sure you won't mind since I'm a huge bigot, right?
[0] https://www.britannica.com/topic/Black-Power-Movement
I've been playing cat and mouse trying to block them for the past week and here are a couple of observations/ideas, in case this is helpful to someone:
* As mentioned above, the bulk of the traffic comes from a large number of IPs, each issuing only a few requests a day, and they pretend to be real UAs.
* Most of them don't bother sending the referrer URL, but not all (some bots from Huawei Cloud do, but they currently don't generate much traffic).
* The first thing I tried was to throttle bandwidth for URLs that contain id= (which on a cgit instance generate the bulk of the bot traffic). So I set the bandwidth to 1Kb/s and thought surely most of the bots will not be willing to wait for 10-20s to download the page. Surprise: they didn't care. They just waited and kept coming back.
* BTW, they also used keep alive connections if ones were offered. So another thing I did was disable keep alive for the /cgit/ locations. Failed that enough bots would routinely hog up all the available connections.
* My current solution is to deny requests for all URLs containing id= unless they also contain the `notbot` parameter in the query string (and which I suggest legitimate users add in the custom error message for 403). I also currently only do this if the referrer is not present but I may have to change that if the bots adapt. Overall, this helped with the load and freed up connections to legitimate users, but the bots didn't go away. They still request, get 403, but keep coming back.
My conclusion from this experience is that you really only have two options: either do something ad hoc, very specific to your site (like the notbot in query string) that whoever runs the bots won't bother adapting to or you have to employ someone with enough resources (like Cloudflare) to fight them for you. Using some "standard" solution (like rate limit, Anubis, etc) is not going to work -- they have enough resources to eat up the cost and/or adapt.
We block China and Russia. DDOS attacks and other hack attempts went down by 95%. We have no chinese users/customers so in theory this does not effect business at all. Also russia is sanctioned and our russian userbase does not actually live in russia, so blocking russia did not effect users at all.
How did you choose where to get the IP addresses to block? I guess I'm mostly asking where this problem (i.e. "get all IPs for country X") is on the scale from "obviously solved" to "hard and you need to play catch up constantly". I did a quick search and found a few databases but none of them looks like the obvious winner.
One starts to wonder, at what point might it be actually feasible to do it the other way around, by whitelisting IP ranges. I could see this happening as a community effort, similar to adblocker list curation etc.
The Pokémon Go company tried that shortly after launch to block scraping. I remember they had three categories of IPs: - Blacklisted IP (Google Cloud, AWS, etc), those were always blocked - Untrusted IPs (residential IPs) were given some leeway, but quickly got to 429 if they started querying too much - Whitelisted IPs (IPV4 addresses are used legitimately by many people), for example, my current data plan tells me m…
The known good list is IPs and ranges I know are good. The known bad list is specific bad actors. The data center networks list is updated periodically based on a list of ASNs belonging to data centers.
There are a lot of problems with using ASNs, even for well-known data center operators. First, they update so often. Second, they often include massive subnets like /13(!), which can apparently overlap with routes announced by other networks, causing false positives. Third, I had been merging networks (to avoid overlaps causing problems in nginx) with something like https://github.com/projectdiscovery/mapcidr but found that it also caused larger overlaps that introduced false positives from adjacent networks where apparently some legitimate users are. Lastly, I had seen suspicious traffic from data center operators like CATO Networks Ltd and ZScaler that are some kind of enterprise security products that route clients through their clouds. Blocking those resulted in some angry users in places I didn't expect...
And none of the accounts for the residential ISPs that bots use to appear like legitimate users https://www.trendmicro.com/vinfo/us/security/news/vulnerabil....
One starts to wonder, at what point might it be actually feasible to do it the other way around, by whitelisting IP ranges. I could see this happening as a community effort, similar to adblocker list curation etc.
The Pokémon Go company tried that shortly after launch to block scraping. I remember they had three categories of IPs: - Blacklisted IP (Google Cloud, AWS, etc), those were always blocked - Untrusted IPs (residential IPs) were given some leeway, but quickly got to 429 if they started querying too much - Whitelisted IPs (IPV4 addresses are used legitimately by many people), for example, my current data plan tells me m…
Wouldn't it be better, if there's an easy way, to just feed such bots shit data instead of blocking them. I know it's easier to block and saves compute and bandwidth, but perhaps feeding them shit data at scale would be a much better longer term solution.
FAFO from both sides. Not defending this bot at all. That said, the shenanigans some rogue or clueless webmasters are up to blocking legitimate and non intrusive or load causing M2M trafic is driving some projects into the arms of 'scrape services' that use far less considerate nor ethical means to get to the data you pay them for. IP blocking is useless if your sources are hundreds of thousands of people worldwide j…
If you feel like you need to do anything at all, I would suggest treating it like any other denial-of-service vulnerability: Fix your server or your application. I can handle 100k clients on a single box, which equates to north of 8 billion daily impressions, and so I am happy to ignore bots and identify them offline in a way that doesn't reveal my methodologies any further than I absolutely have to.