Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Aggressive bots ruined my weekend

herman.bearblog.dev

Post reply on HN
Print view View original

61–70 of 109 posts

Re: Aggressive bots ruined my weekend

#61

Post by ItsBob »

psnehanshu wrote: post #51
ItsBob wrote: post #22

I had a website earlier this year running on Hetzner. It was purely experimenting with some ASP.NET stuff but when looking at the logs, I noticed a shit-load of attempts at various WordPress-related endpoints. I then read something about a guy who deliberately put a honeypot in his robots.txt file. It was pointing to a completely bogus endpoint. Now, the theory was, humans won't read robots.txt so there's no danger,…

What if it was proxied through mobile network on an unsuspecting user's phone? You risk of blocking a whole city or region.

I admit, my approach was rather nuclear but it worked at the time.

I think an evolution would be to use some sort of exponential backoff, e.g. first time offenders get banned for an hour, second time is 4 hours, third time and you're sent into the abyss!

Still crude but fun to play about with.

Re: Aggressive bots ruined my weekend

#62

Post by ItsBob »

bob1029 wrote: post #42
ItsBob wrote: post #22

I had a website earlier this year running on Hetzner. It was purely experimenting with some ASP.NET stuff but when looking at the logs, I noticed a shit-load of attempts at various WordPress-related endpoints. I then read something about a guy who deliberately put a honeypot in his robots.txt file. It was pointing to a completely bogus endpoint. Now, the theory was, humans won't read robots.txt so there's no danger,…

> It's not perfect but it worked for me anyway. This is approximately my approach minus the zip bomb. I use a piece of middleware in my AspNetCore pipeline that tracks logical resource consumption rates per IPv4. If a client trips any of the limits, their IP goes into a HashSet for a period of time. If a client has an IP in this set, they get a simple UTF8 constant string in the response body "You have exceeded resou…

> If a client has an IP in this set, they get a simple UTF8 constant string in the response body "You have exceeded resource limits, please try again later".

Would a simple 429 not do the same thing? You could log repeated 429's and banish accordingly.

Re: Aggressive bots ruined my weekend

#63

Post by seemaze »

dataviz1000 wrote: post #9
VladVladikoff wrote: post #4

Earlier quoted context omitted.

This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of reque…

They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s

That's sleazy. It's slipping drugs into a kids lunchbox and letting smuggle it across the border..

Re: Aggressive bots ruined my weekend

#64

Post by uvaursi »

Chabsff wrote: post #52
uvaursi wrote: post #49

Do we shift over everything to le Dark Web and let the corpos use this one for selling their shit to consumers? These toys don’t want to play nice and there’s no real way to stop them without bringing in things like Real ID and other verifications that infringe on anonymity.

The bots flock to where the data is. Moving to a different network is just begging for the bots to tag along.

We can set different rules on these networks however. We can choose to be choosy at the gate.

Re: Aggressive bots ruined my weekend

#65

Post by alamortsubite »

VladVladikoff wrote: post #10
dataviz1000 wrote: post #9

Earlier quoted context omitted.

They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s

WOW that video! Ain’t no way anyone has EVER read those terms. This feels so insidious that it really should be illegal. Wonder if this exists in the EU or if they have shut it down already?

Yes, and it doesn't matter if they do read the terms- to the average user they sound totally innocuous, especially placed next to a big shiny "GET 500 FREE COINS" button.

Re: Aggressive bots ruined my weekend

#66

Post by antoniojtorres »

curious_curios wrote: post #5
asplake wrote: post #3

> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…

If you have a moderately successful app, sdk or browser extension you will get hit up to add things to it like this. I think most free VPN services also lease out your bandwidth to make their money as well.

This is how so many companies sell from an opaque inventory of “millions” of residential proxies.

Re: Aggressive bots ruined my weekend

#67

Post by alamortsubite »

arethuza wrote: post #60
VladVladikoff wrote: post #10

Earlier quoted context omitted.

WOW that video! Ain’t no way anyone has EVER read those terms. This feels so insidious that it really should be illegal. Wonder if this exists in the EU or if they have shut it down already?

That video has the app asking the user to confirm the use of their device to run a proxy within the app - but is there any hard requirement for this, could apps use this SDK and silently run as a proxy?

My take is it's mostly irrelevant, but read the lobsters post mentioned elsewhere.

Re: Aggressive bots ruined my weekend

#68

Post by nix0n »

immibis wrote: post #27
asplake wrote: post #3

> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…

You can get paid a few dollars (not many) to let them use your connection. I would like Cloudflare's business model (blocking datacenter IPs) to be worthless, so I do it. Haven't tried a withdrawal yet so it could well be a scam. This is not illegal (unless it's a scam).

> This is not illegal

Depends on what they're doing from your connection.

Re: Aggressive bots ruined my weekend

#69

Post by georgefrowny »

intended wrote: post #39
r_singh wrote: post #34

The Internet isn’t possible without scraping. For all the sentiment against scraping public data, doing so remains legal and essential to a lot of the services we use everyday. I think setting guidelines and shaping the web for reduced friction aimed at fair usage rather than turning it political would be the right thing to do.

What ? What do you mean ?

To be fair the heyday of unshit search was driven by mostly-consensual scraping.

Today there are far too many people scraping stuff that isn't intended to be scraped, for profit, and doing it in a heavy-handed way that actually does have a negative and continuous effect on the victim's capacity.

Everyone from AI services too lazy or otherwise unwilling to cache to companies exfiltrating some kind of data for their own commercial purposes.

Re: Aggressive bots ruined my weekend

#70

Post by marcosdumay »

bob1029 wrote: post #42
ItsBob wrote: post #22

I had a website earlier this year running on Hetzner. It was purely experimenting with some ASP.NET stuff but when looking at the logs, I noticed a shit-load of attempts at various WordPress-related endpoints. I then read something about a guy who deliberately put a honeypot in his robots.txt file. It was pointing to a completely bogus endpoint. Now, the theory was, humans won't read robots.txt so there's no danger,…

> It's not perfect but it worked for me anyway. This is approximately my approach minus the zip bomb. I use a piece of middleware in my AspNetCore pipeline that tracks logical resource consumption rates per IPv4. If a client trips any of the limits, their IP goes into a HashSet for a period of time. If a client has an IP in this set, they get a simple UTF8 constant string in the response body "You have exceeded resou…

> they get a simple UTF8 constant string in the response body "You have exceeded resource limits, please try again later"

I imagine they get a 429 response code, but if they don't, you may want to change that.

I do think you are on the right place in that it's important to let those requests get the correct error, so if innocent people are affected, they at least get to see there's something wrong.

Post reply on HN