Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

411–420 of 422 posts

Re: An update on residential proxies and the scraper situation

#411
post #134

Earlier quoted context omitted.

I'm...not old enough to read a Business Insider article about being annoyed by lowercase letters.

I think most young people (with friends to text) have already encountered this.

Most young people are familiar with lowercase sentences and have been for a while, yes

Re: An update on residential proxies and the scraper situation

#412
post #10

Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.

This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.

of those millions of residential proxies i would be surprised if you could find a dozen people who knew this was happening on their devices

Re: An update on residential proxies and the scraper situation

#413

Earlier quoted context omitted.

Because they don't have the informed consent* of the owner of the device wich ends up running the code? * no, small print in a click-through agreement doesn't count.

I think they do actually. It's pretty clear from the consent screen that they're doing what they're doing.

ask any of the people who installed it if they understand what they're doing and especially what the consequences of that are. i doubt you could get a dozen people who do of the millions and millions who installed it.

Re: An update on residential proxies and the scraper situation

#414
post #410

Earlier quoted context omitted.

Anubis appears to be a temporarily-useful stopgap that has been cargo culted into prominence and an expectation of permanent usefulness, for reasons I don't fully understand. The cost of solving the default Anubis PoW is negligible on cloud servers , and it's even lower if you use native code rather than JavaScript to solve it, which Tavis Ormandy helpfully demonstrated last year ( https://lock.cmpxchg8b.com/anubis.h…

> If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse. Then anubis adopts it itself, increases the amount of work that needs to be done and the bar stays the same again for everyone? Seems like mostly a non-issue unless there is an arms race towards ever more optimized solvers which I don't believe is possible.

> > If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.

> Then anubis adopts it itself, increases the amount of work that needs to be done and the bar stays the same again for everyone?

No, the bar doesn't "stay the same" for everyone interacting with Anubis.

My otherwise-perfectly-usable 8-year-old phone, which can't be patched to run a native solver, becomes even more unusable on sites gates with proof-of-work challenges like Anubis.

This is the whole problem with PoW. It forces thousands or millions or billions of client devices to do increasing amounts of useless work which is relatively easy for cloud-based attackers to adapt to, but very difficult for hardware-constrained and software-ossified mobile clients to adapt to.

In other words, it asymmetrically punishes the clients that it's not intending to punish.

Re: An update on residential proxies and the scraper situation

#415

Earlier quoted context omitted.

Proof of work captchas are widely deployed, especially on more niche sites. Kiwiflare is one (used for a harassment forum)

You could've just said Anubis to avoid giving those guys advertisement.

I could’ve said a lot of things. Kiwiflare is the only one I’m aware of (and can remember name of) that provides a visible proof of work occurring in the browser, regardless of reputation scoring (which I don’t think they use). Although, I suppose by definition, anyone using that site is of poor character…

If you can think of another example, please be helpful and provide it, thanks.

Re: An update on residential proxies and the scraper situation

#416
post #79

Earlier quoted context omitted.

Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...

From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.

Also by doing network traffic analysis

Re: An update on residential proxies and the scraper situation

#417
post #183

Earlier quoted context omitted.

> but PoW scales Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest. > becoming much more obvious and easy to block, or they have to use massive amounts of compute. If you believe this, please contact me: I think compute is free[1] and can probably help you out. [1]: https://news.ycombinator.co…

Can you not design a PoW that is most efficient in a browser? Don't brute force hashes like Hashcash/Bitcoin, do something similar to RandomX instead but in JS. Browsers ought to run the fastest JS interpreters already so if interpreting JS becomes the bulk of the work, that attack might not work. Maybe even involve the DOM or whatever else makes sense.

> Browsers ought to run the fastest JS interpreters already

Well they don't.

Users want the website to work sooner, and care little about whether a for-loop of elements take 10ms or 20ms if it only happens once.

JS can be AOT compiled if you can wait a few _seconds_ -- which users don't want, so browsers don't bother.

Our attacker however, rightly observes they only have to pay that compilation cost once.

Re: An update on residential proxies and the scraper situation

#418

Earlier quoted context omitted.

PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".

Most users of residential proxies just get a SOCKS5 address and routing, they don't actually get computational resources of the infected systems beyond that. The user of the proxies, the operator of what the article describes as a control node, would be the device responsible for the PoW. Do you have any evidence that AI providers aren't using residential proxies?

Yes, the status quo right now is merely bandwidth... but if there was money to be made in providing a small amount of compute for the sake of solving the next gen captcha's, you can bet offerings will expand to meet it.

It's impossible to prove a negative. They could all be running secondary scrapers using malware proxies... but what we do have is plenty of evidence of them using fixed IP pools with appropriate user agents. I don't see any Chinese user agents... so two guesses who may be driving the bulk of these AI scraper requests via residential proxies.

Re: An update on residential proxies and the scraper situation

#419

Earlier quoted context omitted.

PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".

Residential proxy bandwidth is extremely expensive, comparatively speaking. It can be up to $1 per GB but is more typically about $0.20 per GB.

Right, this is orthagonal to the discussion though. While the IPs and bandwidth might be free, managing the malware botnet and trying to keep a low profile so as to not attract attention of authorities, makes it a risky market to cater to.

Those rates are still cheaper than some datacentres charge in parts of the world.

Re: An update on residential proxies and the scraper situation

#420

Earlier quoted context omitted.

PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".

If you pop my machine and use it to route 100 MBit/s, I might not notice for months. If I hear the fan spinning at night, you're probably getting caught immediately. If you pop my mom's TV box and use it to route data within the connection's capabilities, you're getting away with it. If you consume a little bit of resources, still. If you consume enough to be useful for these kind of challenges, chances are her TV pl…

If you're tech savvy enough to monitor your equipment, you're probably smart enough not to install "free VPNs" or "free stream pirate apps" to get pwned in the first place. You probably changed your router's default password and kept its firmware up to date, and don't give your TV internet access, right? That's more than Mom has done.
Post reply on HN