Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Aggressive bots ruined my weekend

herman.bearblog.dev

Post reply on HN
Print view View original

41–50 of 109 posts

Re: Aggressive bots ruined my weekend

#41

Post by TimorousBestie »

flaviuspopan wrote: post #35
TimorousBestie wrote: post #6

I think he should consider getting out of the indie blog hosting business. It’s only going to get worse as the internet continues to decay and he can’t be making all that much off the service.

His persistent efforts are the reason I pay for Bear Blog. I think he should fight for the chance to come out on the other side of whatever future we’re heading towards.

I pay for Bear Blog, too. But this year has been problem after problem for its sole proprietor, and I don’t think it’s going to get better.

Re: Aggressive bots ruined my weekend

#42

Post by bob1029 »

ItsBob wrote: post #22

I had a website earlier this year running on Hetzner. It was purely experimenting with some ASP.NET stuff but when looking at the logs, I noticed a shit-load of attempts at various WordPress-related endpoints. I then read something about a guy who deliberately put a honeypot in his robots.txt file. It was pointing to a completely bogus endpoint. Now, the theory was, humans won't read robots.txt so there's no danger,…

> It's not perfect but it worked for me anyway.

This is approximately my approach minus the zip bomb. I use a piece of middleware in my AspNetCore pipeline that tracks logical resource consumption rates per IPv4. If a client trips any of the limits, their IP goes into a HashSet for a period of time. If a client has an IP in this set, they get a simple UTF8 constant string in the response body "You have exceeded resource limits, please try again later".

The other aspect of my strategy is to use AspNetCore (Kestrel). It is so fast that you can mostly ignore the noise as long as things are configured properly and you make reasonable attempts to address the edge case of an asshole trying to break your particular system on purpose. A HashSet as the very first piece of middleware rejecting bad clients is exceedingly efficient. We aren't even into URL routing at this point.

I have found that attempting to catalog and record all of the naughty behavior my web server sees is the highest risk to DDOS so far. Logging lines like "banned client rejected" every time they try to come in the door is shooting yourself in the foot with regard to disk wear, IO utilization, et. al. There is no reason you should be logging all of that background radiation to disk or even thinking about it. If your web server cant handle direct exposure to the hard vacuum of space, it can be placed behind a proxy/CDN (i.e., another web server that doesn't suck).

Re: Aggressive bots ruined my weekend

#43

Post by TYPE_FASTER »

dataviz1000 wrote: post #9
VladVladikoff wrote: post #4

Earlier quoted context omitted.

This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of reque…

They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s

Also see https://www.youtube.com/watch?v=AGaiVApKfmc - "Avoid restrictions and blocks using the fastest and most stable proxy network"...they're pretty upfront with this, aren't they?

Oh, and they will sell you the datasets they've already scraped using mobile devices: https://brightdata.com/lp/web-data/datasets

This actually explains a phishing attack where I received a text from somebody purporting to be a co-worker asking for an Apple gift card. The name was indeed an employee from a different part of the large company I worked for at the time, but LinkedIn was the only possible link I could figure out that was at least somewhat publicly available information.

This should probably be required in all CS curriculum: https://ocw.mit.edu/courses/res-tll-008-social-and-ethical-r...

Re: Aggressive bots ruined my weekend

#44

Post by cuu508 »

dataviz1000 wrote: post #9
VladVladikoff wrote: post #4

Earlier quoted context omitted.

This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of reque…

They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s

IMO Google Play should check apps for presence of this SDK and other similar SDKs, and, upon detection, treat these apps as malware.

Re: Aggressive bots ruined my weekend

#45

Post by uvaursi »

reustle wrote: post #37
kelvinjps10 wrote: post #19

Maybe moving the blog service to completely static and letting cloudfare pages handle it, could help?

Cloudflare is not a solution. Only leading to a further centralized internet.

I think the OP is suggesting using a caching layer at HTTP output, and suggesting CF as an option (a quick/cheap one).

If you have an axe to grind with CF you can take it up with them, but it’s an option. Feel free to suggest others.

Re: Aggressive bots ruined my weekend

#46

Post by kijin »

arethuza wrote: post #36
dataviz1000 wrote: post #9

Earlier quoted context omitted.

They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s

I suspect most people, even when told exactly what the app using that SDK would be doing, wouldn't actually see the potential problems...

Until one day, they get swatted for accessing child porn.

Actually, that might be one way to draw attention to the problem. Sign up to some of these shady "residential proxy" services, and access all sorts of nasty stuff through their IPs until your favorite three-letter agency takes notice.

Re: Aggressive bots ruined my weekend

#47

Post by vpShane »

TimorousBestie wrote: post #6

I think he should consider getting out of the indie blog hosting business. It’s only going to get worse as the internet continues to decay and he can’t be making all that much off the service.

No way. People deserve expression and to have a place that's THEIRS where they can foster a community. Much is learned. Playing battle bots is fun at the sysadmin level (for me), maybe not so much for others, but to have a place where people express themselves, and have THEIR place outside of the walled gardens such as social media, AND they protect it from the bots?

That's the battle, and expression, people, their interests, and their communities are worth fighting for. _ESPECIALLY_ in this day and age where botnets/scrapers are using things such as Infatica to mask themselves as residential IP addresses, and mimicking human behaviors to better avoid bot detection.

There's a war on authenticity, people's authentic works, and the reverse: determining if a user is authentic now adays.

Re: Aggressive bots ruined my weekend

#48

Post by karlshea »

Neil44 wrote: post #23
cupofjoakim wrote: post #13

We feel this at work too. We run a book streaming platform with all books, booklists, authors, narrators and publishers available as standalone web pages for SEO, in the multiple millions. Last 6 months have turned into a hellscape - for a few reasons: 1. It's become commonplace to not respect rate limits 2. Bots no longer identify themselves by UA 3. Bots use VPNs or similar tech to bypass ip rate limiting 4. Bots u…

Same, I have a few hundred Wordpress sites and bot activity has ramped up a lot over the last year or two. AI scrapers can be quite aggressive and often generate a ton of requests where for example a site has a lot of parameters, the bot will go nuts seeming to iterate through all possible parameters. Sometimes I dig in and try to think of new rules to block the bulk, but I am also wary of AI replacing Google and not…

A client of mine had this exact problem with faceted search, and putting the site behind Fastly didn’t help since you can’t cache millions of combinations. And they don’t have the budget for more than one origin server. The solution was if you’ve got “bot” in your UA Fastly’s VCL returns a 403 with any facet query param. Problem solved. And it’s not going to break anything, all of the information is still accessible to all of the indexers on the actual product pages.

The facet links already had “nofollow” on them, now I’m just enforcing it.

Re: Aggressive bots ruined my weekend

#49

Post by uvaursi »

Do we shift over everything to le Dark Web and let the corpos use this one for selling their shit to consumers? These toys don’t want to play nice and there’s no real way to stop them without bringing in things like Real ID and other verifications that infringe on anonymity.

Re: Aggressive bots ruined my weekend

#50

Post by piggg »

kaoD wrote: post #7
asplake wrote: post #3

> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…

There's crap like https://hola.org/ https://hola.org/legal/sdk https://hola.org/legal/sla > How is it free? > > In return for free usage of Hola Free VPN Proxy, Hola Fake GPS location and Hola Video Accelerator, you may be a peer on the Bright Data network. By doing so you agree to have read and accepted the terms of service of the Bright Data SDK SLA ( https://bright-sdk.com/eula ). You may opt out by becoming a Pre…

There's also a ton of companies selling "make money off your unused internet" apps which are all over tiktok and basically turn yourself into a residential proxy/sketch VPN egress node.

On top of that - lots of free tv/movie streaming stuff that also makes yourself a proxy/egress node. Sometimes you find it on tv/movie streaming devices sold online where it's already loaded on when it arrives.

Post reply on HN