Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

221–230 of 422 posts

Re: An update on residential proxies and the scraper situation

#221

Post by inigyou »

BLKNSLVR wrote: post #151

> There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time. Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same addr…

Why would anyone who doesn't have a use for a residential proxy have an argument for residential proxies?

I use them to scrape closed sites to make the information more open. For example YouTube.

Re: An update on residential proxies and the scraper situation

#222

Post by inigyou »

Catloafdev wrote: post #80
rao-v wrote: post #74

I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands. Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use! EDIT: Lwn.net is perhaps not a…

If it weren't a real problem, these types of articles and services wouldn't exist.

Plenty of complaints exist about things that are not real problems.

Re: An update on residential proxies and the scraper situation

#223

Post by inigyou »

TZubiri wrote: post #78
rao-v wrote: post #74

I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands. Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use! EDIT: Lwn.net is perhaps not a…

Why the air quotes? Evading a ban and using (potentially ill gotten) residential ips to circumvent that refusal of service, is a bad actor.

Surely that depends on the motivation for the ban.

Re: An update on residential proxies and the scraper situation

#224

Post by inigyou »

teravor wrote: post #122

I find the notion that you would use residential proxies to scrape LWN somewhat laughable, I'm reading this article using a VPN. residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).

Right. This whole controversy makes no sense as a pure scraping thing. It seems more like someone is trying to take the web offline.

Re: An update on residential proxies and the scraper situation

#225

Post by aiiotnoodle »

miki123211 wrote: post #207
freehorse wrote: post #200

Earlier quoted context omitted.

Imo the worst is recaptcha. At least with cloudflare the work you have to provide is minimal. With recaptcha it can take me much longer than 5 seconds, and lately I have trouble even completing their challenges correctly. Nowadays if I see a (recaptcha) captcha I drop the site unless I must visit it for some reason, it is not worth the time, the effort or the annoyance.

Most CF / Recaptcha problems are users going "off the golden path", and not realizing that their config changes are at fault. If you're on a consumer router, using a mainstream stock browser with stock settings (maybe plus uBlock Origin), with your Google account logged in, it's very, very likely to just work. If you're part of the .01% of users with opinions about that sort of thing... you're not worth optimizing fo…

A lot of users also run cheap cracked fire sticks and other low reputation hardware that's proxying their residential traffic for nefarious reasons which makes all the big providers put up their guard.

Re: An update on residential proxies and the scraper situation

#226

Post by inigyou »

aorth wrote: post #178
TurdF3rguson wrote: post #51

Earlier quoted context omitted.

What exactly should be illegal here? Scraping websites? AI agents? Not following robots.txt?

The excessive scraping and ignoring robots.txt only breaks the informal social contract established over the past decades of the open internet. The real problem is the companies offering money to developers if they include unrelated SDKs in their calculator or flashlight (for example) applications. Those SDKs add functionality to incorporate those devices into a network that can be used for scraping. The traffic is l…

On what grounds would it be illegal though? Things don't become illegal just because you don't like them. They may become illegal just because the president doesn't like them, but I don't think you're him, and in the absence of that, there has to be a majority of Congress and most of them want a reason.

Re: An update on residential proxies and the scraper situation

#227

Post by inigyou »

TZubiri wrote: post #77

>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on. The first argu…

Residential proxy users don't have the ability to run compute on their proxies.

Re: An update on residential proxies and the scraper situation

#228

Post by tgsovlerkhgsel »

jimmydorry wrote: post #160
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".

If you pop my machine and use it to route 100 MBit/s, I might not notice for months.

If I hear the fan spinning at night, you're probably getting caught immediately.

If you pop my mom's TV box and use it to route data within the connection's capabilities, you're getting away with it. If you consume a little bit of resources, still. If you consume enough to be useful for these kind of challenges, chances are her TV playback will start to stutter, which will be resolved by taking the compromised TV box, and removing the malware using advanced mechanical means called "a trash compactor".

Re: An update on residential proxies and the scraper situation

#229

Post by freehorse »

miki123211 wrote: post #207
freehorse wrote: post #200

Earlier quoted context omitted.

Imo the worst is recaptcha. At least with cloudflare the work you have to provide is minimal. With recaptcha it can take me much longer than 5 seconds, and lately I have trouble even completing their challenges correctly. Nowadays if I see a (recaptcha) captcha I drop the site unless I must visit it for some reason, it is not worth the time, the effort or the annoyance.

Most CF / Recaptcha problems are users going "off the golden path", and not realizing that their config changes are at fault. If you're on a consumer router, using a mainstream stock browser with stock settings (maybe plus uBlock Origin), with your Google account logged in, it's very, very likely to just work. If you're part of the .01% of users with opinions about that sort of thing... you're not worth optimizing fo…

At least for me, CF is fine; recaptcha is the only one I really have problems with.

I dont care what recaptcha wants to optimize for. I dont think that using a vpn is that a rare thing anyway. If others have figured out how to do it without requiring spending 30 seconds to solve a captcha, I dont see why websites still use recaptcha/captchas for that.

And that it is "my fault" not being logged into google I was least expecting to see here.

Re: An update on residential proxies and the scraper situation

#230

Post by solid_fuel »

everfrustrated wrote: post #14
corbet wrote: post #9

Earlier quoted context omitted.

Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.

Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.

Caches only help for pages that have been requested recently. The behavior of crawlers - going from one page to the next across the whole site - will probably not be mitigated significantly by a cache.
Post reply on HN