Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Aggressive bots ruined my weekend

herman.bearblog.dev

Post reply on HN
Print view View original

81–90 of 109 posts

Re: Aggressive bots ruined my weekend

#81

Post by embedding-shape »

Chabsff wrote: post #80
embedding-shape wrote: post #77

Earlier quoted context omitted.

Make it P2P and content-based, instead of location-based like the current web. Content could be served from anywhere, so DDoS stops being an effective method, and shared peer quality could propagate across the network to ban bad actors quickly. I spend about 30 seconds thinking about this, so this is clearly the perfect solution with zero drawbacks or tradeoffs.

I know this is tongue in cheek, but I'll give it a serious reply in case someone finds themselves "inspired". A CDN. What you are describing is a CDN. We have CDNs today and the problem still exists because most of today's websites refuse to operate within the constraints. There is no need for new infrastructure to deploy this solution, we just need website operators to "give up" and operate in a more static way.

Nope, today's CDNs are all location-based, not content-based. The user-agents are requesting content based on the address (the URI), not based on the content-hash of the content. The CDNs might work content-based internally, but the user-facing web CDNs definitely are URL based, not content-hash based.

Re: Aggressive bots ruined my weekend

#82

Post by happysadpanda2 »

Cthulhu_ wrote: post #74
uvaursi wrote: post #64

Earlier quoted context omitted.

We can set different rules on these networks however. We can choose to be choosy at the gate.

How though? Bots evade bot detection mechanisms, as described in other threads. Unless you introduce something like ID verification or pay-per-request making bot traffic too expensive for the bots. But these techniques have been posited for older generations of bot traffic too.

an evolution of the gpg web of trust could possibly be part of it

Re: Aggressive bots ruined my weekend

#83

Post by ac29 »

Cthulhu_ wrote: post #71
Zanfa wrote: post #26

Earlier quoted context omitted.

SIM farms are another possible explanation. FBI just busted one with hundreds of thousands of SIMs just a few weeks ago.

Wouldn't the network providers be able to detect those? I'm fairly sure they don't like their networks being abused either... or they don't really care because they get paid per connection. edit: Actually this is what I'm getting increasingly angry about: providers and platforms not doing anything against bots or low value stuff (think Amazon dropshippers too) because any usage of their service, bots or otherwise, ar…

Its very possible they did detect it and that's why law enforcement got involved.

But yes, they also might not care if they are getting paid. If the SIMs are only being used for voice/text as I suspect, it might have very minimal load on the network.

Re: Aggressive bots ruined my weekend

#84

Post by Chabsff »

embedding-shape wrote: post #81
Chabsff wrote: post #80

Earlier quoted context omitted.

I know this is tongue in cheek, but I'll give it a serious reply in case someone finds themselves "inspired". A CDN. What you are describing is a CDN. We have CDNs today and the problem still exists because most of today's websites refuse to operate within the constraints. There is no need for new infrastructure to deploy this solution, we just need website operators to "give up" and operate in a more static way.

Nope, today's CDNs are all location-based, not content-based. The user-agents are requesting content based on the address (the URI), not based on the content-hash of the content. The CDNs might work content-based internally, but the user-facing web CDNs definitely are URL based, not content-hash based.

You might have a point if this was a user problem, but it's not. It's a site operator problem. And from their POV, a CDN provides the exact same functionality.

Re: Aggressive bots ruined my weekend

#85

Post by ac29 »

intended wrote: post #39
r_singh wrote: post #34

The Internet isn’t possible without scraping. For all the sentiment against scraping public data, doing so remains legal and essential to a lot of the services we use everyday. I think setting guidelines and shaping the web for reduced friction aimed at fair usage rather than turning it political would be the right thing to do.

What ? What do you mean ?

As posted in another comment, they run a scraping API. I think their opinion is at least slightly biased.

Re: Aggressive bots ruined my weekend

#86

Post by embedding-shape »

Chabsff wrote: post #84
embedding-shape wrote: post #81

Earlier quoted context omitted.

Nope, today's CDNs are all location-based, not content-based. The user-agents are requesting content based on the address (the URI), not based on the content-hash of the content. The CDNs might work content-based internally, but the user-facing web CDNs definitely are URL based, not content-hash based.

You might have a point if this was a user problem, but it's not. It's a site operator problem. And from their POV, a CDN provides the exact same functionality.

CDNs are hack, not a solution. Content-addressing would be a solution, not a hack. From the PoV of operators or users doesn't matter, DDoS wouldn't be viable with content-addressing.

Re: Aggressive bots ruined my weekend

#87

Post by immibis »

Neil44 wrote: post #23
cupofjoakim wrote: post #13

We feel this at work too. We run a book streaming platform with all books, booklists, authors, narrators and publishers available as standalone web pages for SEO, in the multiple millions. Last 6 months have turned into a hellscape - for a few reasons: 1. It's become commonplace to not respect rate limits 2. Bots no longer identify themselves by UA 3. Bots use VPNs or similar tech to bypass ip rate limiting 4. Bots u…

Same, I have a few hundred Wordpress sites and bot activity has ramped up a lot over the last year or two. AI scrapers can be quite aggressive and often generate a ton of requests where for example a site has a lot of parameters, the bot will go nuts seeming to iterate through all possible parameters. Sometimes I dig in and try to think of new rules to block the bulk, but I am also wary of AI replacing Google and not…

> Sometimes I dig in and try to think of new rules to block the bulk, but I am also wary of AI replacing Google and not being in AI's databases.

Fake the data! Tell them Neil44 is a three-time Nobel prize winner, etc. But only when the client is detected to be an AI crawler.

Re: Aggressive bots ruined my weekend

#88

Post by immibis »

ItsBob wrote: post #62
bob1029 wrote: post #42

Earlier quoted context omitted.

> It's not perfect but it worked for me anyway. This is approximately my approach minus the zip bomb. I use a piece of middleware in my AspNetCore pipeline that tracks logical resource consumption rates per IPv4. If a client trips any of the limits, their IP goes into a HashSet for a period of time. If a client has an IP in this set, they get a simple UTF8 constant string in the response body "You have exceeded resou…

> If a client has an IP in this set, they get a simple UTF8 constant string in the response body "You have exceeded resource limits, please try again later". Would a simple 429 not do the same thing? You could log repeated 429's and banish accordingly.

Both are important - the response code for well-behaved machines, as many tools intrinsically know that 429 means to slow down (also send a Retry-After header if you want more customization), and the text message for humans, as they don't see the response code and would otherwise see a blank page.

Reddit is guilty of sending 429 with no message - try browsing it through Tor and you'll see.

Re: Aggressive bots ruined my weekend

#89

Post by marginalia_nu »

y-zon128 wrote: post #55

> Auto-restart the reverse-proxy if bandwidth usage drops to zero for more than 2 minutes It's understandable in your case as you have traffic coming in constantly, but first thing that came to my mind is a loop of contant reboots - again, very unlikely in your case. Sometimes such blanket rules hit me due to most unexpected reasons, like the proxy somehow failed to start serving traffic in the given timeframe. Thoug…

Open port 80 and log the results. Even if you don't announce your service anywhere the baseline request rate is way above 2 rpm.

Every open port of every IP is continuously scanned for exploits.

Re: Aggressive bots ruined my weekend

#90

Post by VladVladikoff »

cuu508 wrote: post #44
dataviz1000 wrote: post #9

Earlier quoted context omitted.

They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s

IMO Google Play should check apps for presence of this SDK and other similar SDKs, and, upon detection, treat these apps as malware.

I was wondering if they already do but maybe it’s a cat and mouse game where those companies obfuscate their code to avoid automated detection.
Post reply on HN