Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disrupting the largest residential proxy network

cloud.google.com

Post reply on HN
Print view View original

191–200 of 230 posts

Re: Disrupting the largest residential proxy network

#191

Post by moduspol »

kawsper wrote: post #83
whartung wrote: post #6

My understanding is that routing through residential IPs is a part of the business of some VPN providers. I don't know how above board they are on this (as in notifying customers that this may happen, however buried in the usage agreement, or even allowing them to opt out). But, my main point, is that the whole business is "on the up and up" vs some dark botnet.

Oxylabs sells proxies for scrapers, I suppose you can use the socks-proxy as a VPN, and they claim to use Honeygain. Honeygain is a platform where people sell their residential internet connection and bandwidth to these companies for money. For comparison Honeygain pays someone 10 cents per GB, and Oxylabs sells it for $8/GB.

We need a better market. I'd sell for $7/GB.

Re: Disrupting the largest residential proxy network

#193

Post by Rasbora »

I've helped multiple people remove residential proxy malware that was turning their network into a brightdata exit node and they had no idea / did not consent to it. Why is google selectively targeting one provider while letting others operate freely?

You can check if your network is infected here: https://layer3intel.com/is-my-network-a-residential-proxy

Re: Disrupting the largest residential proxy network

#194

Post by Aurornis »

megous wrote: post #164
Aurornis wrote: post #30

Earlier quoted context omitted.

> I want exactly the same content visible no matter who you are or where you are connecting from The reason those IP addresses get blocked is not because of "who" is connecting, but "what" Traffic from datacenter address ranges to sites like Reddit is almost entirely bots and scrapers. They can put a tremendous load on your site because many will try to run their queries as fast as they can with as many IPs as they c…

Okay. So what does ten million requests cost, then? Like... a dollar? Is it a dollar? Is it two dollars if they splurge? Because if the deterrent here is a line item so small it shows up as 'miscellaneous vibes' on a balance sheet, that's not a barrier. That's a tip jar.

Residential proxies are often priced by Gigabyte and the pricing is several orders of magnitude higher than your estimate.

Re: Disrupting the largest residential proxy network

#195

Post by JDye »

Craighead wrote: post #148
bigiain wrote: post #121

Earlier quoted context omitted.

Am I the only one cynically thinking that "Russia, Iran, DPRK, PRC, etc" is the "But think of the chiiildren!!!" excuse for doing this? And when Google say "IPIDEA’s proxy infrastructure is a little-known component of the digital ecosystem leveraged by a wide array of bad actors." What they really mean is " ... leveraged by actors indiscriminately scraping the web and ignoring copyright - that are not us." I can't he…

No, what they're saying is what they said, what you're implying reveals a strange bias. Web scraping through residential proxies? Please think through your thoughts more. There's much more effective and efficient ways to do so. Multiple bad actors, like ransomware affiliates, have been caught using residential proxy networks. But by all means, don't let facts and cyber threat intelligence get in the way.

Residential proxies aren't used for scraping? That doesn't align well with my experience...

Re: Disrupting the largest residential proxy network

#196

Post by vlovich123 »

xhcuvuvyc wrote: post #124
vlovich123 wrote: post #120

Earlier quoted context omitted.

> Some users may knowingly install this software on their devices, lured by the promise of “monetizing” their spare bandwidth. Sounds like they’re targeting networks even if the users are ok participating in, precisely what you’re saying is ok. As for malware enrolling people into the network, it depends if the operator is doing it or if the malware is 3rd parties trying to get a portion of the cash flow. In the latt…

> These SDKs, which are offered to developers across multiple mobile and desktop platforms, surreptitiously enroll user devices into the IPIDEA network. ?

Here’s an alternate spin

> These SDKs, which are offered to developers across multiple mobile and desktop platforms.

> other actors then surreptitiously enroll user devices into the IPIDEA network using these frameworks.

I’m not saying Google did the wrong thing, but it is one private entity essentially handing out a death sentence on its own. The only mitigating thing is that a) technical disruptions were either on their own infra b) legal judgements they then enforced with cooperation from others like Cloudflare. But it’s not clear what the legal proceedings were actually like

Re: Disrupting the largest residential proxy network

#197

Post by 7thpower »

walletdrainer wrote: post #137
ghxst wrote: post #133

Earlier quoted context omitted.

They do KYC when you want to unblock certain domains.

Also not my experience, even though I’ve had to email them for whitelisting. It might just be because my account is very old?

Maybe, or more likely you’re not trying to pull in content that is considered high risk to them, such as YouTube transcripts.

Re: Disrupting the largest residential proxy network

#198

Post by tonymet »

mrweasel wrote: post #166
edg5000 wrote: post #118

Residential proxies are the only way to crawl and scrape. It's ironic for this article to come from the biggest scraping company that ever existed! If you crawl at 1Hz per crawled IP, no reasonable server would suffer from this. It's the few bad apples (impatient people who don't rate limit) who ruin the internet for both users and hosters alike. And then there's Google.

First of: Google has not once crashed one of our sites with GoogleBot. They have never tried to by-pass our caching and they are open and honest about their IP ranges, allowing us to rate-limit if needed. The residential proxies are not needed, if you behave. My take is that you want to scrape stuff that site owners do not want to give you and you don't want to be told no or perhaps pay a license. That is the only ca…

You can’t get much crawling done from published cloud IPs. Residential proxies are the only way to do most crawls today.

That said, I support Google working to shut these networks down, since they are almost universally bad.

It’s just a shame that there’s no where to go for legitimate crawling activities.

Re: Disrupting the largest residential proxy network

#200

Post by digiown »

ErroneousBosh wrote: post #172
Ronsenshi wrote: post #129

Earlier quoted context omitted.

One thing about Google is that many anti-scraping services explicitly allow access to Google and maybe couple of other search engines. Everybody else gets to enjoy CloudFlare captcha, even when doing crawling at reasonable speeds. Rules For Thee but Not for Me

Why are you scraping sites in the first place? What legitimate reason is there for you doing that?

Dunno, building a Google competitor? How do you think Google got started?
Post reply on HN