Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Aggressive bots ruined my weekend

herman.bearblog.dev

Post reply on HN
Print view View original

21–30 of 109 posts

Re: Aggressive bots ruined my weekend

#21

Post by wat10000 »

VladVladikoff wrote: post #4
asplake wrote: post #3

> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…

This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of reque…

Lately Reddit has been showing me posts in subreddits for some of these services. They pitch "passive income" by sharing your connection, an easy way to make a few bucks by renting out your unused capacity. What happens is that you become an endpoint for their shady VPNs. These subreddits are full of people complaining that they're getting hit by abuse complaints from their ISPs. Naturally, these services claim to forbid any nefarious activity, and naturally they don't actually care.

Re: Aggressive bots ruined my weekend

#22

Post by ItsBob »

I had a website earlier this year running on Hetzner. It was purely experimenting with some ASP.NET stuff but when looking at the logs, I noticed a shit-load of attempts at various WordPress-related endpoints.

I then read something about a guy who deliberately put a honeypot in his robots.txt file. It was pointing to a completely bogus endpoint. Now, the theory was, humans won't read robots.txt so there's no danger, but bots and the like will often read robots.txt (at least to figure out what you have... they'll ignore the "deny" for the most part!) and if they try and go to that fake endpoint you can be 100% sure (well, as close as possible) that it's not a human and you can ban them.

So I tried that.

I auto-generated a robots.txt file on the fly. It was cached for 60 seconds or so as I didn't want to expend too many resource on it. When you asked for it, you either got the cached one or I created a new one. The CPU-usage was negligible.

However, I changed the "deny" endpoint each time I built the file in case the baddies cached it, however, it still went to the same ASP.NET controller method. By hitting it, I sent a 10GB zip bomb and your IP was automatically added to the FW block list.

It was quite simple: anyone that hit that endpoint MUST be dodgy... I believe I even had comments for the humans that stumbled across it letting them know that if they went to this endpoint in their browser it was an automatic addition to the firewall blocklist.

Anyway... at first I caught a shit load of bad guys. There were thousands at first and then the numbers dropped and dropped to only tens per day.

Anyway, this is a single data point but for me, it worked... I have no regrets about the zip bomb either :)

I have another site that I'm working on so I may evolve it a bit so that you are banned for a short time and if you come back to the dodgy endpoint then I know you're a bot so into the abyss with you!

It's not perfect but it worked for me anyway.

Re: Aggressive bots ruined my weekend

#23

Post by Neil44 »

cupofjoakim wrote: post #13

We feel this at work too. We run a book streaming platform with all books, booklists, authors, narrators and publishers available as standalone web pages for SEO, in the multiple millions. Last 6 months have turned into a hellscape - for a few reasons: 1. It's become commonplace to not respect rate limits 2. Bots no longer identify themselves by UA 3. Bots use VPNs or similar tech to bypass ip rate limiting 4. Bots u…

Same, I have a few hundred Wordpress sites and bot activity has ramped up a lot over the last year or two. AI scrapers can be quite aggressive and often generate a ton of requests where for example a site has a lot of parameters, the bot will go nuts seeming to iterate through all possible parameters. Sometimes I dig in and try to think of new rules to block the bulk, but I am also wary of AI replacing Google and not being in AI's databases.

Re: Aggressive bots ruined my weekend

#24

Post by nurettin »

corbet wrote: post #18
VladVladikoff wrote: post #4

Earlier quoted context omitted.

This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of reque…

The "compliance officer" at Bright Data, instead, offered me a special deal to protect my site from their bots ... they run a protection racket along with all the rest of their nastiness.

I worked for an Amazon scraping business and they used Luminati (Now Brightdata) for a few months until I figured out a way to avoid the ban hammer and got rid of their proxy.

They indeed provided "high quality" residential and cellular ips and "normal quality" data center ips. You had to keep cycling the ip pool every 2-3 days which cost extra. It felt super shady. It isn't their bots, they lease connections to whoever is paying, and they don't care what people do in there.

Re: Aggressive bots ruined my weekend

#26

Post by Zanfa »

asplake wrote: post #3

> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…

SIM farms are another possible explanation. FBI just busted one with hundreds of thousands of SIMs just a few weeks ago.

Re: Aggressive bots ruined my weekend

#27

Post by immibis »

asplake wrote: post #3

> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…

You can get paid a few dollars (not many) to let them use your connection. I would like Cloudflare's business model (blocking datacenter IPs) to be worthless, so I do it. Haven't tried a withdrawal yet so it could well be a scam. This is not illegal (unless it's a scam).

Re: Aggressive bots ruined my weekend

#28

Post by add-sub-mul-div »

sudosays wrote: post #12
TimorousBestie wrote: post #6

I think he should consider getting out of the indie blog hosting business. It’s only going to get worse as the internet continues to decay and he can’t be making all that much off the service.

Counterpoint: I think he should stay and fight the good fight. Indie blog businesses are great for the health of the human internet, and I don't think surrendering preemptively will help things get better.

That's an easy thing to say if it's someone else's time that's being wasted and not your own. But there may not be a path back to the internet under which this project was conceived.

It could be like staying on Twitter and Reddit after their respective declines. You're only suffering an opportunity cost for your own time and preventing the internet from evolving better alternatives.

Re: Aggressive bots ruined my weekend

#29

Post by immibis »

ItsBob wrote: post #22

I had a website earlier this year running on Hetzner. It was purely experimenting with some ASP.NET stuff but when looking at the logs, I noticed a shit-load of attempts at various WordPress-related endpoints. I then read something about a guy who deliberately put a honeypot in his robots.txt file. It was pointing to a completely bogus endpoint. Now, the theory was, humans won't read robots.txt so there's no danger,…

It's interesting to study, right? This is the Internet equivalent of background radiation. Harmless in most cases. Exploit scanners aren't new to the LLM age and shouldn't overload your server - unless you're vulnerable to the exploit.

Fun fact: Some people learn about new exploits by watching their incoming requests.

Re: Aggressive bots ruined my weekend

#30

Post by jrochkind1 »

cupofjoakim wrote: post #13

We feel this at work too. We run a book streaming platform with all books, booklists, authors, narrators and publishers available as standalone web pages for SEO, in the multiple millions. Last 6 months have turned into a hellscape - for a few reasons: 1. It's become commonplace to not respect rate limits 2. Bots no longer identify themselves by UA 3. Bots use VPNs or similar tech to bypass ip rate limiting 4. Bots u…

I hate relying on a proprietary single-source product from a company I don't particularly trust, but (free) Cloudflare Turnstile works for me, only thing I've found that does.

I only protect certain 'dangerous/expensive' (accidentally honeypot-like) paths in my app, and can leave the stuff I actually want crawlers to get, and in my app that's sufficient.

It's a tension because yeah I want crawlers to get much of my stuff for SEO (and don't want to give a monopoly to Google on it either, i want well-behaved crawlers I've never heard of to have access to it too. But not at the cost of resources i can't afford).

Post reply on HN