Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

151–160 of 422 posts

Re: An update on residential proxies and the scraper situation

#151

Post by BLKNSLVR »

> There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.

Maybe there's no point for the scanned server to block the address, but couldn't collective / shared block lists help with sites that may get scanned by the same address after the initial one?

The main problem becomes managing lists of millions of individual addresses. My (only semi-reliable these days, due to lack of time for maintenance) little project has nearly 2.3 million addresses recorded - although only 590k are from 2026, and only 38 were probes on ports 80 and 443. So maybe more manageable than I thought (but my servers don't host anything beyond personal interest to me, and access is filtered via cloudflare, which is it's own "internet control issue").

> In general, these companies range from those that aspire toward some appearance of legitimacy, advertising "GDPR compliance" for example, to others that are just overtly sleazy.

Overall, my gut feel on residential proxies is that they're an untrustworthy scourge. I'd be interested in any arguments for residential proxies by people who don't (intend to) profit from using it facilitating them.

In regards to Bright Data, one of the companies that attempts to appear legitimate, at minimum these domains should be blocked:

brdtnet.com

luminatinet.com

bright-sdk.com

luminati.io

As listed in this article, on HN's front page 34 days ago: https://news.ycombinator.com/item?id=48422993 (https://blog.includesecurity.com/2026/06/the-smart-tv-in-you...)

Re: An update on residential proxies and the scraper situation

#152

Post by Chu4eeno »

BoorishBears wrote: post #130
llbbdd wrote: post #119

Earlier quoted context omitted.

> forced lowercase ??? shift is an extra key to press

https://www.businessinsider.com/lowercase-typing-altman-dors... most shibboleths are subtle like that

aseigo was the OG, «aseigo: the triumphs and travails of a shift-key-challenged KDE hacker» was his blog byline a couple of decades ago.

Re: An update on residential proxies and the scraper situation

#153

Post by ArtTimeInvestor »

In theory, proof of work that is used to mine a cryptocurrency could be a solution.

Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads.

The question is which resources typical consumer hardware has that large centralized compute power does not. In-browser POW to pay websites would only be possible if such a resource exists.

I am not familiar with the topic, but maybe CPU power and memory? Both seem significant in a typical consumer device.

Napkin math: If a consumer device can generate $100 per month, that would be 100/30/24/60/60=$0.00004 per second. If the user waits for 5 seconds before the first pageview, that would then make the website provider $0.0002 per visitor. Serving a million visitors per month is nowadays easily possible on a $10/month machine. So the $0.0002x1000000 = $200 would make the website a nice profit.

Re: An update on residential proxies and the scraper situation

#154

Post by [deleted] »

Groxx wrote: post #79
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...

[deleted]

Re: An update on residential proxies and the scraper situation

#155

Post by karlgkk »

ArtTimeInvestor wrote: post #153

In theory, proof of work that is used to mine a cryptocurrency could be a solution. Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads. The question is which resources typical consumer hardwar…

Proof of work captchas are widely deployed, especially on more niche sites. Kiwiflare is one (used for a harassment forum)

Re: An update on residential proxies and the scraper situation

#156

Post by charcircuit »

throwaway7356 wrote: post #141
charcircuit wrote: post #109

>types of operator running residential-proxy networks to attack web sites. This is such a malicious interpretation. Do you think VPN operating are also trying to attack websites? Both offer the same kind of product. >paid for hijacking their users' network connections Nothing is being hijacked. Again the author is using wording to try and paint these people as malicious actors. >Recently, LWN was subjected what was,…

> Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors. If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network. It doesn't seem like monopoly abuse to exclude such malware from applica…

>sell access to the user's internal residential network

That is not what the SDK was doing. The actual code in the SDK protects against this (simplified to take less space):

    if (addr.isSiteLocalAddress() || addr.isLoopbackAddress()) {
        LogUtils.e("PopaTunnelAsyncThread", "Hacking? The Host Resolved Ip is " + addr + " on tunnel id:" + tunnelId);
        throw new IllegalArgumentException("Hacking? The tunnel host resolved ip is internal");
    }
Local and loopback addresses like 10.0.0.0, 172.16.0.0, 192.168.0.0, and 127.0.0.0 do not work. It will not connect to people's private file servers on their network.

Re: An update on residential proxies and the scraper situation

#157

Post by zarzavat »

This is a predictable consequence of age verification laws and social media bans. Formerly VPNs were a nice to have but now they are a necessity in many countries to navigate the modern internet.

The cheapest way to get a VPN (and if you're a horny and broke teenager perhaps the only way) is to trade your clean but censored IP address for an uncensored IP address in another country. You accept the bot traffic in return, or externalize it to your parents or the owner of the internet connection.

Re: An update on residential proxies and the scraper situation

#158

Post by entropyneur »

Sorry, I understand scraping is a problem, but talking about open Internet while simultaneously complaining you can no longer discriminate datacenter IPs like you used to is hypocrisy.

I use a datacenter-based IPv6 address because my local ISPs don't offer v6 connectivity and the Internet is already broken for me. And generally the entire idea of a "residential" IP address smells.

Re: An update on residential proxies and the scraper situation

#159

Post by selfhoster1312 »

zarzavat wrote: post #157

This is a predictable consequence of age verification laws and social media bans. Formerly VPNs were a nice to have but now they are a necessity in many countries to navigate the modern internet. The cheapest way to get a VPN (and if you're a horny and broke teenager perhaps the only way) is to trade your clean but censored IP address for an uncensored IP address in another country. You accept the bot traffic in retu…

That does not explain why so many residential VPNs operate with so many IPs in countries where there are no social media bans. Here in France:

- i know many people who buy shady IPTV boxes from stores/markets for like 50€/year

- i know some people who use "smart lightbulbs" and other nonsense

- almost everyone i know plays free smartphone games, which as LWN reminded, may contain a shady SDK

Re: An update on residential proxies and the scraper situation

#160

Post by jimmydorry »

harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
Post reply on HN