Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

231–240 of 422 posts

Re: An update on residential proxies and the scraper situation

#231

Post by throwaway7356 »

charcircuit wrote: post #156
throwaway7356 wrote: post #141

Earlier quoted context omitted.

> Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors. If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network. It doesn't seem like monopoly abuse to exclude such malware from applica…

>sell access to the user's internal residential network That is not what the SDK was doing. The actual code in the SDK protects against this (simplified to take less space): if (addr.isSiteLocalAddress() || addr.isLoopbackAddress()) { LogUtils.e("PopaTunnelAsyncThread", "Hacking? The Host Resolved Ip is " + addr + " on tunnel id:" + tunnelId); throw new IllegalArgumentException("Hacking? The tunnel host resolved ip i…

It'll still connect to IPv6 addresses and bypass any firewalls.

Also users might become part (victim?) of a police investigation because of illegal actions that seem to originate from their local residential connection.

So still good to take down such backdoors. Would be nice to go after the botnet operators as well...

Re: An update on residential proxies and the scraper situation

#232

Post by solid_fuel »

noxvilleza wrote: post #62
TurdF3rguson wrote: post #49

Earlier quoted context omitted.

Yes I've seen it. ClaudeBot will gleefully announce itself when it hammers my niche website a million times a day.

At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.

Blocking is too obvious. I would prefer to feed back false information but only to LLM crawlers.

Re: An update on residential proxies and the scraper situation

#233

Post by fc417fc802 »

alightsoul wrote: post #106
Groxx wrote: post #79

Earlier quoted context omitted.

Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...

From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.

Supposedly, but not really. I regularly encounter sites where cloudflare serves me with an ambiguous ban notice rather than a proof of work. What's worse is that these apparent IP bans take effect even if I already had a valid active session (ie previously passed the check).

Yes, a VPN involved. That doesn't make it okay and notice that anubis by default works without issue (though possibly with a more difficult challenge) in the exact same scenario.

Re: An update on residential proxies and the scraper situation

#234

Post by Cakez0r »

miki123211 wrote: post #207
freehorse wrote: post #200

Earlier quoted context omitted.

Imo the worst is recaptcha. At least with cloudflare the work you have to provide is minimal. With recaptcha it can take me much longer than 5 seconds, and lately I have trouble even completing their challenges correctly. Nowadays if I see a (recaptcha) captcha I drop the site unless I must visit it for some reason, it is not worth the time, the effort or the annoyance.

Most CF / Recaptcha problems are users going "off the golden path", and not realizing that their config changes are at fault. If you're on a consumer router, using a mainstream stock browser with stock settings (maybe plus uBlock Origin), with your Google account logged in, it's very, very likely to just work. If you're part of the .01% of users with opinions about that sort of thing... you're not worth optimizing fo…

If you don't live in a first world country then you will also find yourself on the "too bad, don't care" list.

Re: An update on residential proxies and the scraper situation

#236

Post by solid_fuel »

rao-v wrote: post #74

I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands. Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use! EDIT: Lwn.net is perhaps not a…

I don't think people sit around going "Grrrr who can I ban next?". Instead this stuff gets noticed because you see the webserver at 99% CPU utilization for 2 days straight, check the logs, and see you are somehow getting crawled by half the IPs in New York City.

Re: An update on residential proxies and the scraper situation

#237

Post by fc417fc802 »

PlasmaPower wrote: post #107
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck. Then again, a large portion of the problem seems to be bots making way too many requests and in general not being…

If that happens the browser engines (all what, 3 of them?) can add a PoW API to call into native code. Or a pathologically scalar algorithm can be adopted so that wasm is good enough. RandomX or something close to it probably qualifies.

Re: An update on residential proxies and the scraper situation

#238

Post by CodesInChaos »

TZubiri wrote: post #77

>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on. The first argu…

> The moment it does you'll be able to tell because Bitcoin will crash to 0.

The "workaround" for PoW is running the PoW computation on hardware that's better suited for the task. Bitcoin mining has been using ASIC for many years now.

Let's say a legitimate user is willing to wait for one minute on a budget phone. Then your PoW is limited to what that phone can compute in one minute. But on the attacker's specialized hardware this computation only costs fractions of a penny, so they are barely hindered by it.

The SHA256 based PoW scheme has a very heavy ASIC advantage. People have tried to design PoW scheme that minimize the custom hardware advantage, but I'm not sure if they managed to close the gap far enough to make PoW feasible for this application.

Re: An update on residential proxies and the scraper situation

#239

Post by Sesse__ »

anonym29 wrote: post #202
mootothemax wrote: post #138

Earlier quoted context omitted.

Interesting. How do they tell the difference between legitimate and forged ip owner records?

It's not about traffic identification at all, but rather a hashing algorithm that is deliberately resistant to parallelization and GPU/ASIC acceleration, which shrinks the gap in solving speed between the fastest systems (i.e. datacenter-class compute resources) and typical systems (e.g. the CPU in your smartphone or laptop).

Uh, is it resistant to parallelization across multiple sites? Because that's the situation for the scrapers. They're not trying to solve a single PoW challenge across many cores.

Re: An update on residential proxies and the scraper situation

#240

Post by RickHull »

inigyou wrote: post #210
Gigachad wrote: post #167

Earlier quoted context omitted.

I looked this up and realised it’s the page I’d seen briefly on a range of websites lately. It’s not annoyed me at all. Not nearly as much as having to complete captchas with slow refreshing tiles.

A fun fact about Google captchas: they've often decided whether you will succeed or fail the captcha before you do the captcha.

This seems nonsensical. Care to elaborate?
Post reply on HN