Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

321–330 of 422 posts

Re: An update on residential proxies and the scraper situation

#321

Post by RetroTechie »

phendrenad2 wrote: post #289

Ever since bots became a problem on the internet 10-20 years ago, it has seemed like the common-sense solution is some kind of micropayment. Pay $0.01 to view the page. When money is on the line, scrapers are likely to be more well-behaved, even if they do pay. The problem is, and has always been, the friction of payment. How do you pay $0.01? The credit card processors will tack on a $6 surcharge. We need a trusted…

> I'm guessing there's some egregious regulation that makes micropayments impossible.

More likely there isn't any kind of universal standard that's easy to implement for browser makers, has low overhead, and preserves internet users' anonymity as much as possible.

The currently existing friction of using micropayments is the problem here, I suspect.

Re: An update on residential proxies and the scraper situation

#322

Post by armchairhacker »

Sesse__ wrote: post #282
armchairhacker wrote: post #176

Earlier quoted context omitted.

PoW can theoretically scale effectively infinite because it can mine cryptocurrency. Millions of compromised IoT devices hitting your server? Now you have enough money for a faster server. It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.

This is called “installing a cryptominer on your web page” and is generally considered illegitimate.

> generally considered illegitimate

But why? Obviously an unjustified cryptominer is bad, like unnecessarily slow JavaScript, but this one has a good purpose and to the user is no different than PoW.

Re: An update on residential proxies and the scraper situation

#323

Post by fc417fc802 »

alightsoul wrote: post #249
fc417fc802 wrote: post #233

Earlier quoted context omitted.

Supposedly, but not really. I regularly encounter sites where cloudflare serves me with an ambiguous ban notice rather than a proof of work. What's worse is that these apparent IP bans take effect even if I already had a valid active session (ie previously passed the check). Yes, a VPN involved. That doesn't make it okay and notice that anubis by default works without issue (though possibly with a more difficult chal…

There's lists of data center IPs. You're probably in them and That's why you're getting banned

Regardless of the precise logic it's no excuse for the policy. Simply hand out a sufficiently difficult PoW to prevent widespread abuse.

I'm quite certain it isn't a generic "datacenter" list though because a given VPN exit that was working will suddenly stop. Meanwhile I have a valid cookie yet that is disregarded.

Re: An update on residential proxies and the scraper situation

#326

Post by kps »

pocksuppet wrote: post #302
asdfsa32 wrote: post #108

Earlier quoted context omitted.

HN is surprisingly very very guilty of a whole lot of anti-user patterns and behaviour that other companies get regularly lamented. Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.

Poor accessibility Good.

HN accessibility isn't bad, since it's all just text.

Re: An update on residential proxies and the scraper situation

#327

Post by simoncion »

BoorishBears wrote: post #90
Bratmon wrote: post #81

Earlier quoted context omitted.

I actually read a really interesting article from a relatively small blog explaining that they're receiving a massive amount of scraper traffic from residential proxies. The article was called "The one we're commenting on"

> The LWN content-management system contains over 750,000 items (articles, comments, security alerts, etc) dating back to the adoption of the "new" site code in 2002. We still have, in our archives, everything we did in the over four years we operated prior to the change as well. In addition, the mailing-list archives contain many hundreds of thousands of emails. Does that sound like your typical self-hosted blog?

> Does that sound like your typical self-hosted blog?

When compared to

> They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.

yes, absolutely.

Re: An update on residential proxies and the scraper situation

#328

Post by Barbing »

jolmg wrote: post #118
Barbing wrote: post #100

Earlier quoted context omitted.

And even if you spent a minute explaining the proposition to a user off the street, it still wouldn't be fair unless you laid out the drawbacks. Which leads me to a question. There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?

> There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.

Thank you. Gmail must not be like our fellow HN users we see here, quoting a couple:

  “I'm tiny and only run little personal stuff. I just block vast IP address blocks.”

  “Apologies. :( Since you say you've never visited the website before, then that means you're either in one of the countries or in one of the residential IP ranges that I've had to block.”
Although Google isn’t afraid to completely block iCloud private relay from Google scholar. Other sites may reject the first iCloud private relay visit, then reopen site in new tab and often automatically assigns new unblocked IP. Anyway, I would’ve thought it’s an acceptable cost from e.g. Google’s perspective to block ranges that did something bad once in spite of collateral damage.

Re: An update on residential proxies and the scraper situation

#329

Post by solatic »

georgyo wrote: post #276

The article at the end talks about how is very easy for arbitrary apps from app stores can install a residential proxy on your phone. 10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what. And both ios and android make it hard to deny apps network access. But really, this finally explains the hordes of real…

> 10 years ago, apps had to explicitly state if they needed network access. And then the powers that be decided that really all apps need network access no matter what.

Why does network access need to be a binary, all or nothing?

When you install an app, the app should request permissions to specific DNS names, i.e. pointing to the servers that the app's authors operate. If I install Todoist, the app should only ask for access to Todoist's servers. If I install Netflix, the app should only ask for access to Netflix's servers. The OS can then put a DNS firewall in and block any network access that wasn't granted when the app was installed.

> And both ios and android make it hard to deny apps network access

The list of apps that genuinely need "any" network access (web browsers, VPN apps, stuff like Termux...) is incredibly small compared to the list of apps that need access to a small number of VPN targets (these days, most apps). Apple / Google could even decide, if they really want to make it easy for apps to request network access, to basically allow apps to automatically get network access, so long as the list of domains the app needs access to is no more than a handful. The security value of isolating "all" network access permissions to only the relative handful of apps that actually need to request it, would be huge.

Re: An update on residential proxies and the scraper situation

#330

Post by seff »

asdfsa32 wrote: post #108
duskwuff wrote: post #96

Earlier quoted context omitted.

> Unfortunately whatever HN is using routinely blocks my login with "Sorry." I believe that's the HN application itself, not a WAF in front of it.

HN is surprisingly very very guilty of a whole lot of anti-user patterns and behaviour that other companies get regularly lamented. Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.

I think you just described most of the reason I like it. Plain text sites tend to scale text, allow pinch to zoom, and are just generally faster.

Post deletion we've seen used for manipulation on other sites, it's a sticky topic. HN seems to want you to stand by your word, so it makes sense here.

Post reply on HN