Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

351–360 of 422 posts

Re: An update on residential proxies and the scraper situation

#351

Post by TurdF3rguson »

robinsonb5 wrote: post #265
inigyou wrote: post #226

Earlier quoted context omitted.

On what grounds would it be illegal though? Things don't become illegal just because you don't like them. They may become illegal just because the president doesn't like them, but I don't think you're him, and in the absence of that, there has to be a majority of Congress and most of them want a reason.

Because they don't have the informed consent* of the owner of the device wich ends up running the code? * no, small print in a click-through agreement doesn't count.

I think they do actually. It's pretty clear from the consent screen that they're doing what they're doing.

Re: An update on residential proxies and the scraper situation

#352

Post by inigyou »

crote wrote: post #345
jsnell wrote: post #208

Earlier quoted context omitted.

Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users. > The dilemma for bots: when tokens are bound to the connecting ip, sc…

The success of the ongoing Anubis rollout proves the opposite. People are used to slowly-loading websites - the rise of garbage SPAs has seen to that. Staring at a spinner for a second every once in a while is not an issue for genuine users. On the other hand, the additional CPU usage rises the compute cost of scraping by several orders of magnitude. If you don't have to scrape this specific website, you'd be stupid…

Scrapers generally aren't looking for random websites to scrape. They have a specific URL in mind. Only if the goal was DDoS would they not care which URL was accessed.

Re: An update on residential proxies and the scraper situation

#353

Post by inigyou »

crote wrote: post #349
jazzyjackson wrote: post #190

Earlier quoted context omitted.

Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.

How is that not a massive GDPR violation?

You can just break the law, if you don't get caught, especially if you're rich.

Re: An update on residential proxies and the scraper situation

#354

Post by inigyou »

Aurornis wrote: post #310
miki123211 wrote: post #209

Earlier quoted context omitted.

Oh, but you can PoW every page. Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).

> Oh, but you can PoW every page. They meant you can’t PoW every page transition. If clicking every link on your website throws you back to another Anubis page for 2-3 seconds, users will bounce. That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.

There are major websites that take more than 2-3 seconds to load every new page. Google, for example. Reddit. Facebook. Instagram. Doesn't seem to have hurt them.

Re: An update on residential proxies and the scraper situation

#355

Post by inigyou »

harshreality wrote: post #165
mootothemax wrote: post #137

Earlier quoted context omitted.

> The anubis author has stated they recognize it's an arms race, but PoW scales. The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you. The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will toler…

Anubis's default 1-week token lifetime may not be nearly enough to dissuade enough scraper networks to make a difference, particularly with the default weight->difficulty level hierarchy, but that's for individual site admins to determine. We can all argue based on how we envision "ideal" scraper networks being run and whether the web-PoW concept would stand up to that. However, what matters at present is that anubis…

One of the ideas behind Anubis was to incentivize a scraper to stop hiding, because every change of identity brings another challenge page.

Re: An update on residential proxies and the scraper situation

#356

Post by inigyou »

ezst wrote: post #123
cyberax wrote: post #93

Earlier quoted context omitted.

Hah. I have a homelab with a couple of sites, including a personal Forgejo installation. Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough. However, this time they hammered my computer for 12 hours with about 200 requests per…

The paradox of them selling "intelligence on demand" or "coding agents rivaling the best developers" and yet having dumb as fart bots/scrapers is lost on many. But not all.

There's no paradox, just deception. "AI" is dumb as a bag of rocks, but good at convincing people that it isn't. However, it's also a really good semantic vector search that can find and combine existing answers to many pre-asked questions.

Re: An update on residential proxies and the scraper situation

#358

Post by inigyou »

xena wrote: post #348
dlenski wrote: post #346

As this article points out, it's tremendously unclear who is using residential proxies . The big AI models claim they're not using them. I'm not inclined to "just believe them", but no incriminating evidence has leaked, and—as pointed out in the article—many of the bots that are running on these residential proxy botnets are coded in incredibly stupid and inefficient ways. How confident are people who research this s…

That's the only theory we have that doesn't sound like a conspiracy theory. The only other credible ideas are that someone's doing some kind of dataset arbitrage by scraping the fuck out of everything and selling companies data that is technically new (by means of the scrape date being newer).

Or someone is trying to DdoS the entire web.

Re: An update on residential proxies and the scraper situation

#359

Post by inigyou »

dannyfritz07 wrote: post #281
georgyo wrote: post #276

The article at the end talks about how is very easy for arbitrary apps from app stores can install a residential proxy on your phone. 10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what. And both ios and android make it hard to deny apps network access. But really, this finally explains the hordes of real…

GrapheneOS allows you to deny network access per app pretty trivially. Google Play services make it a bit more difficult because the app might marshall the network request through that; I'm not sure how to verify that behavior when it happens.

I think Google Play Services will only marshal data to and from Google. This will bypass the permission if you're using it as a firewall, but won't let the app run a residential proxy.

Re: An update on residential proxies and the scraper situation

#360

Post by inigyou »

BLKNSLVR wrote: post #168
oasisbob wrote: post #126

Earlier quoted context omitted.

As someone dealing with similar on a large site, I'd love to see a private community to discuss some of these issues.

Disclaimer: this works for my very small number of personal services that I run. I have no idea how this would (or probably wouldn't) scale at all. Also, the methodology I describe below is based on what I'm able to do technically, which is pretty much limited to bash scripting. On my external-most device I have a firewall that logs addresses that attempt to connect to ports behind which there are no services, and th…

Scrapers and port scanners are mostly entirely separate activities. If you don't have any unexpected open ports you have nothing to fear from a port scanner, and blocking them won't stop you from getting attacked by scrapers.
Post reply on HN