Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

211–220 of 422 posts

Re: An update on residential proxies and the scraper situation

#211

Post by inigyou »

jimmydorry wrote: post #160
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".

Residential proxy bandwidth is extremely expensive, comparatively speaking. It can be up to $1 per GB but is more typically about $0.20 per GB.

Re: An update on residential proxies and the scraper situation

#212

Post by inigyou »

andai wrote: post #45
mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

It isn't. AI scraping has nothing to do with it, for the reasons you said. Someone wants the web to go offline, they are DDoSing the entire web, and it's working. For some reason we are tackling the symptom instead of finding out who that person is. Come on, it can't be that hard to subpoena Bright Data. The law enforcement system knows how to track down someone who's trying to be anonymous on the internet.

Re: An update on residential proxies and the scraper situation

#213

Post by inigyou »

BLKNSLVR wrote: post #168
oasisbob wrote: post #126

Earlier quoted context omitted.

As someone dealing with similar on a large site, I'd love to see a private community to discuss some of these issues.

Disclaimer: this works for my very small number of personal services that I run. I have no idea how this would (or probably wouldn't) scale at all. Also, the methodology I describe below is based on what I'm able to do technically, which is pretty much limited to bash scripting. On my external-most device I have a firewall that logs addresses that attempt to connect to ports behind which there are no services, and th…

Most residential users change their IP address every 24 hours.

Re: An update on residential proxies and the scraper situation

#214

Post by inigyou »

Loic wrote: post #162
ArtTimeInvestor wrote: post #153

In theory, proof of work that is used to mine a cryptocurrency could be a solution. Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads. The question is which resources typical consumer hardwar…

Proof of work, even "custom", where the user does not need a particular interaction with the page, does not work. The scrapers are running headless Chrome and solving the work. They do not care, they do not pay the bill, the compromised system's owner pays the bill. I have such system for the registration form on one of my website to prevent the double validation of emails to be used to spam emails of victims. The Po…

Residential proxy users don't get to run computation on the proxies.

Re: An update on residential proxies and the scraper situation

#215

Post by inigyou »

karlgkk wrote: post #155
ArtTimeInvestor wrote: post #153

In theory, proof of work that is used to mine a cryptocurrency could be a solution. Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads. The question is which resources typical consumer hardwar…

Proof of work captchas are widely deployed, especially on more niche sites. Kiwiflare is one (used for a harassment forum)

You could've just said Anubis to avoid giving those guys advertisement.

Re: An update on residential proxies and the scraper situation

#216

Post by inigyou »

noxvilleza wrote: post #62
TurdF3rguson wrote: post #49

Earlier quoted context omitted.

Yes I've seen it. ClaudeBot will gleefully announce itself when it hammers my niche website a million times a day.

At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.

If you block it, it comes back with a residential proxy network and headless Chrome. Better not to block based on the obvious signs.

Re: An update on residential proxies and the scraper situation

#217

Post by inigyou »

Bratmon wrote: post #10

Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.

And it's made necessary because another group of people thought that selling IP blocking services would be a good idea. One party sells walls, another party sells ladders.

Well, one party gives away free walls if you agree to fill your castle with surveillance cameras you don't control.

Re: An update on residential proxies and the scraper situation

#218

Post by inigyou »

Barbing wrote: post #100
not-a-llm wrote: post #60

Earlier quoted context omitted.

by consent they mean a dialog/EULA with careful wording was display and the user clicked ok.

And even if you spent a minute explaining the proposition to a user off the street, it still wouldn't be fair unless you laid out the drawbacks. Which leads me to a question. There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?

There are absolutely no actual drawbacks for most users.

Re: An update on residential proxies and the scraper situation

#219

Post by inigyou »

Bratmon wrote: post #59
BoorishBears wrote: post #24

Earlier quoted context omitted.

Thank god for residential proxies. Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet

By providing a way for corporate AI scrapers to operate with impunity and force the last few independently-run websites to move to the cloud?

By providing a way for independent hackers to extract data from closed commercial websites.

Re: An update on residential proxies and the scraper situation

#220

Post by nikolife2016 »

Running a small public JSON API, the traffic breakdown is eye-opening. Roughly half is trust/uptime "scanners" and generic monitors; a solid chunk is well-behaved crawlers that declare themselves with real UAs and honor robots; and then there's a long tail of vuln-scanners blindly probing for /.env, /.git, wp-login and the like. The genuinely evasive residential-proxy scraping is a minority by volume but by far the hardest to separate from real users — it's the one bucket where UA and IP both look residential, so you can't tell bot from human without behavioral signals. What's shifted in the last year: the "polite" bots got politer, while the abusive layer moved almost entirely onto residential proxies. IP reputation alone is basically dead as a filter now.
Post reply on HN