Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

341–350 of 422 posts

Re: An update on residential proxies and the scraper situation

#341

Post by [deleted] »

fc417fc802 wrote: post #237
PlasmaPower wrote: post #107

Earlier quoted context omitted.

I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck. Then again, a large portion of the problem seems to be bots making way too many requests and in general not being…

If that happens the browser engines (all what, 3 of them?) can add a PoW API to call into native code. Or a pathologically scalar algorithm can be adopted so that wasm is good enough. RandomX or something close to it probably qualifies.

[deleted]

Re: An update on residential proxies and the scraper situation

#342

Post by jay_kyburz »

Symbiote wrote: post #120
jay_kyburz wrote: post #15

Earlier quoted context omitted.

How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.

I have essentially this at work, but the scrapers ignore it. (Or at least many, many scrapers ignore it.)

It wont work unless everybody does it because otherwise it's more work for the scrapers not less. They need to implement two systems instead of one.

And we'll never get everybody to do it.

Re: An update on residential proxies and the scraper situation

#343

Post by jolmg »

Barbing wrote: post #328
jolmg wrote: post #118

Earlier quoted context omitted.

> There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.

Thank you. Gmail must not be like our fellow HN users we see here, quoting a couple: “I'm tiny and only run little personal stuff. I just block vast IP address blocks.” “Apologies. :( Since you say you've never visited the website before, then that means you're either in one of the countries or in one of the residential IP ranges that I've had to block.” Although Google isn’t afraid to completely block iCloud private…

> Anyway, I would’ve thought it’s an acceptable cost from e.g. Google’s perspective to block ranges that did something bad once in spite of collateral damage.

Almost nobody is on a static IP. ISPs have an interest to keep their customers connected and not affected by other customers, so they'd probably use as large address pools as possible for their dynamic IP rotations. "Did something bad once" is short-lived unless address ranges are large.

Tech companies like Google are interested in being global monopolies to dominate markets and also having as good and large-sampled statistical data as possible from everyone. Blocking thousands of users for each bad user that can likely easily switch to another block if they really want to doesn't seem effective. It would also affect their reputation. Who wants to rely on an email service that may block you because of other users on your IP block? Having a reputation for being reliable is important for businesses.

With small personal sites/blogs like the ones you mentioned, they can just block most of the planet without a problem. There's not as much reason to be as reachable as possible (some may be satisfied with an audience of a handful of people), and they're also more budget conscious.

In the case of Google blocking an Apple service, they not blocking users; they're blocking Apple. The users can just not use Apple's service.

Re: An update on residential proxies and the scraper situation

#344

Post by anonym29 »

Sesse__ wrote: post #266
anonym29 wrote: post #253

Earlier quoted context omitted.

Typically, the machine doing the content processing, including solving PoW, is the centralized "control" node described in the article, not the machines who's IP addresses are being used. In typical residential proxy networks, the residential proxies are exposed to the customer (the person paying for and using the proxies) as just SOCKS5 addresses, and no computational power from those compromised devices is made ava…

That's a completely different question, your claim was about parallelism.

Ah, I see what you're getting at. Yes. You can think of any given computer aa having a fixed amount of compute budget for these types of acceleration-resistant hashing algorithms. Let's say the scraper can perform 10,000 hashing operations per second total on their machine, and needs an average of 1,000 hashing operations to solve the PoW. It's a minor detail, but note that these PoW challenges non-deterministically vary in the number of hashing operations needed to produce a valid hash, not dissimilar to bitcoin mining, where a hash with a certain number of 0s prefixed is sought, and the scraper essentially has to brute force through all possible inputs until an input that produces a valid hash is found.

In a well-designed PoW systems, there is a per-site prefix or suffix that is required to be prepended or appended to these random inputs, and it may change not only between websites, but even between PoW sessions on the same website, and should not be predictable - only being disclosed to the client at the time the PoW challenge is issued. In such a case, the scraper cannot simply precompute a bunch of valid hashes that work across multiple sites, nor a bunch of valid hashes that will always be good for even one site, the scraper operator will need to compute these hashes (with a limited budget to do so) upon initiating each PoW session.

Re: An update on residential proxies and the scraper situation

#345

Post by crote »

jsnell wrote: post #208
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users. > The dilemma for bots: when tokens are bound to the connecting ip, sc…

The success of the ongoing Anubis rollout proves the opposite. People are used to slowly-loading websites - the rise of garbage SPAs has seen to that. Staring at a spinner for a second every once in a while is not an issue for genuine users.

On the other hand, the additional CPU usage rises the compute cost of scraping by several orders of magnitude. If you don't have to scrape this specific website, you'd be stupid not to move on and hit someone else.

Ideally the general cost of scraping would be high enough that it isn't affordable any more - especially anonymized - but considering the amount of money brainlessly being pumped into AI I doubt that'll happen any time soon.

You could of course also make the argument that the user's time is worth something as well and should be included as part of the cost, but that ship sailed a loooong time ago. If you care about that, you should be calling for the death of client-side Javascript and any form of advertising.

Re: An update on residential proxies and the scraper situation

#346

Post by dlenski »

As this article points out, it's tremendously unclear who is using residential proxies.

The big AI models claim they're not using them. I'm not inclined to "just believe them", but no incriminating evidence has leaked, and—as pointed out in the article—many of the bots that are running on these residential proxy botnets are coded in incredibly stupid and inefficient ways.

How confident are people who research this stuff that the RP botnets are actually being used for AI training?

Re: An update on residential proxies and the scraper situation

#347

Post by crote »

dannyfritz07 wrote: post #284
out_of_protocol wrote: post #270

Earlier quoted context omitted.

> It trades something fungible and incredibly cheap (CPU) it could be RAM-bound, which is very much NOT cheap nowadays :)

Yes, but the people with the RAM nowadays are the data centers, not the end users.

Sure, but end users as a group still have a significant amount of RAM.

Even on a low-specced machine you can afford to have the currently-active website tab consume a few hundreds of megabytes of RAM. It was mostly sitting idle anyways, so most people aren't even going to notice. Multiply that by a few thousand concurrent visitors and you're burning hundreds of gigabytes without anyone caring.

Some scraper being forced to install hundreds of gigabytes of extra RAM in their crawling node? They will notice having to go from a $0.50 / h instance to a $15.00 / h one, solely for the extra RAM requirement.

Re: An update on residential proxies and the scraper situation

#348

Post by xena »

dlenski wrote: post #346

As this article points out, it's tremendously unclear who is using residential proxies . The big AI models claim they're not using them. I'm not inclined to "just believe them", but no incriminating evidence has leaked, and—as pointed out in the article—many of the bots that are running on these residential proxy botnets are coded in incredibly stupid and inefficient ways. How confident are people who research this s…

That's the only theory we have that doesn't sound like a conspiracy theory. The only other credible ideas are that someone's doing some kind of dataset arbitrage by scraping the fuck out of everything and selling companies data that is technically new (by means of the scrape date being newer).

Re: An update on residential proxies and the scraper situation

#349

Post by crote »

jazzyjackson wrote: post #190
alightsoul wrote: post #106

Earlier quoted context omitted.

From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.

Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.

How is that not a massive GDPR violation?

Re: An update on residential proxies and the scraper situation

#350

Post by crote »

Aurornis wrote: post #310
miki123211 wrote: post #209

Earlier quoted context omitted.

Oh, but you can PoW every page. Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).

> Oh, but you can PoW every page. They meant you can’t PoW every page transition. If clicking every link on your website throws you back to another Anubis page for 2-3 seconds, users will bounce. That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.

> That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.

Except that doing hundreds of requests from the same IP makes it pretty trivial to detect scraping, opening you up to being banned, or fun stuff like a slowloris or being fed poisoned data.

Post reply on HN