Aggressive bots ruined my weekend
herman.bearblog.dev
Aggressive bots ruined my weekend
1–10 of 109 posts
Re: Aggressive bots ruined my weekend
#2[flagged]
Re: Aggressive bots ruined my weekend
#3> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers.
Wild indeed, and potentially horrific for the owners of the affected devices also! Any corroboration for that out there?
Re: Aggressive bots ruined my weekend
#4> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…
This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc.
Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of requests from their bots. The funny part is the didn’t even stop, after acknowledging the complaint.
Re: Aggressive bots ruined my weekend
#5> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…
If you have a moderately successful app, sdk or browser extension you will get hit up to add things to it like this. I think most free VPN services also lease out your bandwidth to make their money as well.
Re: Aggressive bots ruined my weekend
#6I think he should consider getting out of the indie blog hosting business. It’s only going to get worse as the internet continues to decay and he can’t be making all that much off the service.
Re: Aggressive bots ruined my weekend
#7> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…
There's crap like https://hola.org/
> How is it free? > > In return for free usage of Hola Free VPN Proxy, Hola Fake GPS location and Hola Video Accelerator, you may be a peer on the Bright Data network. By doing so you agree to have read and accepted the terms of service of the Bright Data SDK SLA (https://bright-sdk.com/eula). You may opt out by becoming a Premium user.
This "VPN" is what powers these residential proxies: https://brightdata.com/
I'm sure there are many other companies like this.
Re: Aggressive bots ruined my weekend
#8You may want to take a look at Pingoo (https://github.com/pingooio/pingoo), a reverse proxy with automatic TLS that can also block bots with advanced rules that go beyond simple IP blocking.
Re: Aggressive bots ruined my weekend
#9> What's wild is that these scrapers rotate through thousands of IP addresses during their scrapes, which leads me to suspect that the requests are being tunnelled through apps on mobile devices, since the ASNs tend to be cellular networks. I'm still speculating here, but I think app developers have found another way to monetise their apps by offering them for free, and selling tunnel access to scrapers. Wild indeed,…
This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of reque…
They provide an SDK for mobile developers. Here is a video of how it works. [0]
Re: Aggressive bots ruined my weekend
#10Earlier quoted context omitted.
This is actually a commonly known fact. There are many services now that sell “residential proxies”, which are always mobile IP addresses. Since mobile IPs use CGNat it’s also not great to block the IP because it can be like geofencing an entire city or town. Some examples are: oxylabs, iproyal, brightdata, etc. Recently I filed an abuse complaint directly with brightdata because I was getting hit with 1000s of reque…
They provide an SDK for mobile developers. Here is a video of how it works. [0] [0] https://www.youtube.com/watch?v=1a9HLrwvUO4&t=15s
WOW that video! Ain’t no way anyone has EVER read those terms. This feels so insidious that it really should be illegal. Wonder if this exists in the EU or if they have shut it down already?