Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

201–210 of 422 posts

Re: An update on residential proxies and the scraper situation

#201

Post by anonym29 »

jimmydorry wrote: post #160
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".

Most users of residential proxies just get a SOCKS5 address and routing, they don't actually get computational resources of the infected systems beyond that. The user of the proxies, the operator of what the article describes as a control node, would be the device responsible for the PoW.

Do you have any evidence that AI providers aren't using residential proxies?

Re: An update on residential proxies and the scraper situation

#202

Post by anonym29 »

mootothemax wrote: post #138
ammario wrote: post #128

Earlier quoted context omitted.

There are PoW approaches that even the playing field between data centers and desktops. RandomX is my favorite.

Interesting. How do they tell the difference between legitimate and forged ip owner records?

It's not about traffic identification at all, but rather a hashing algorithm that is deliberately resistant to parallelization and GPU/ASIC acceleration, which shrinks the gap in solving speed between the fastest systems (i.e. datacenter-class compute resources) and typical systems (e.g. the CPU in your smartphone or laptop).

Re: An update on residential proxies and the scraper situation

#203

Post by gamesieve »

BoorishBears wrote: post #65
Bratmon wrote: post #59

Earlier quoted context omitted.

By providing a way for corporate AI scrapers to operate with impunity and force the last few independently-run websites to move to the cloud?

No one's firing up a residential proxy to read your blog, and the corporate AI scrapers have all the resources in the world even without residential proxies They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.

The Bright Data mentioned in the article, as well as other similarly malicious but even harder to identify parties, most certainly do fire up residential proxies, no matter what the site, no matter how useless or duplicate the data which they're trying to get. Not at first - they start by trying to get your content from cheap data center connections - but as soon as some kind of bot-mitigation appears, they move to residential proxies to try and evade that, with a first tier coming from "global south" residential proxies, and then scaling up to (presumably more expensive / less widely available) proxies from the USA (I've seen some from Europe, but very few in relative terms). Each tier also appears to have the option to run JavaScript.

Re: An update on residential proxies and the scraper situation

#204

Post by potamic »

jazzyjackson wrote: post #190
alightsoul wrote: post #106

Earlier quoted context omitted.

From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.

Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.

I doubt they are doing it. You just have to get on a VPN to and see yourself being flooded with captchas despite browsing the web like a normal human and solving dozens of captchas along the way.

Re: An update on residential proxies and the scraper situation

#205

Post by 2000UltraDeluxe »

sixtyj wrote: post #8

The issue with scrapping is the intensity and volume of bots. I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it. Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.

> I think that nobody would care if I use wget or curl for few pages

If only you were the only one doing it...

Re: An update on residential proxies and the scraper situation

#206

Post by sunnybeetroot »

BLKNSLVR wrote: post #168
oasisbob wrote: post #126

Earlier quoted context omitted.

As someone dealing with similar on a large site, I'd love to see a private community to discuss some of these issues.

Disclaimer: this works for my very small number of personal services that I run. I have no idea how this would (or probably wouldn't) scale at all. Also, the methodology I describe below is based on what I'm able to do technically, which is pretty much limited to bash scripting. On my external-most device I have a firewall that logs addresses that attempt to connect to ports behind which there are no services, and th…

It’s a great idea but not sure for larger websites when these residential proxy platforms are using innocent user ip addresses. Then you’re left blocking innocent users. It’s a tough call.

Re: An update on residential proxies and the scraper situation

#207

Post by miki123211 »

freehorse wrote: post #200
miyuru wrote: post #182

Earlier quoted context omitted.

For me Cloudflare is worse, it takes more than 5 seconds, where as anubius take 1-2 secs. funny with all the IP information they have, cloudflare cannot do a better job. (I am on IPv6) and most of the time, its on marketing product pages like in framework main site, which can be cached.

Imo the worst is recaptcha. At least with cloudflare the work you have to provide is minimal. With recaptcha it can take me much longer than 5 seconds, and lately I have trouble even completing their challenges correctly. Nowadays if I see a (recaptcha) captcha I drop the site unless I must visit it for some reason, it is not worth the time, the effort or the annoyance.

Most CF / Recaptcha problems are users going "off the golden path", and not realizing that their config changes are at fault.

If you're on a consumer router, using a mainstream stock browser with stock settings (maybe plus uBlock Origin), with your Google account logged in, it's very, very likely to just work. If you're part of the .01% of users with opinions about that sort of thing... you're not worth optimizing for.

Re: An update on residential proxies and the scraper situation

#208

Post by jsnell »

harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users.

> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.

There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.

Re: An update on residential proxies and the scraper situation

#209

Post by miki123211 »

mootothemax wrote: post #137
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

> The anubis author has stated they recognize it's an arms race, but PoW scales. The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you. The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will toler…

Oh, but you can PoW every page.

Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).

Re: An update on residential proxies and the scraper situation

#210

Post by inigyou »

Gigachad wrote: post #167
Groxx wrote: post #79

Earlier quoted context omitted.

Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...

I looked this up and realised it’s the page I’d seen briefly on a range of websites lately. It’s not annoyed me at all. Not nearly as much as having to complete captchas with slow refreshing tiles.

A fun fact about Google captchas: they've often decided whether you will succeed or fail the captcha before you do the captcha.
Post reply on HN