An update on residential proxies and the scraper situation

Post by fragmede » Sun, Jul 12, 2026, 12:03 PM UTC

dredmorbius wrote: post #278

fragmede wrote: post #251

Earlier quoted context omitted.

Between me and you, I don't think Google would register that amongst the noise of just running the service anyway on their monitoring systems. You're right to point out that if you're trying to get the contents of dead objects the API is of no use though.

Google might not register it, but in some related testing, even a few hundred requests through the API (serial, not concurrent) takes much longer than a single "Past" page request would, even if the latter were substantially rate-limited. Of course, if those requests (to HN itself rather than the API) are blocked entirely, that's a moot issue. On dead/flagged items, there's some value. Whilst the title/URL context ar…

Post by [deleted] » Sun, Jul 12, 2026, 3:31 PM UTC

Hackbraten wrote: post #396

solidasparagus wrote: post #103

Earlier quoted context omitted.

This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.

Software that implements residential proxy networks is always covert and never consensual. It doesn't matter if page 42 of the Terms and Condition "clearly states it." This does not constitute informed consent.

Post by tremon » Sun, Jul 12, 2026, 8:05 PM UTC

crote wrote: post #349

jazzyjackson wrote: post #190

Earlier quoted context omitted.

Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.

How is that not a massive GDPR violation?

Post by BLKNSLVR » Sun, Jul 12, 2026, 9:26 PM UTC

inigyou wrote: post #360

BLKNSLVR wrote: post #168

Earlier quoted context omitted.

Disclaimer: this works for my very small number of personal services that I run. I have no idea how this would (or probably wouldn't) scale at all. Also, the methodology I describe below is based on what I'm able to do technically, which is pretty much limited to bash scripting. On my external-most device I have a firewall that logs addresses that attempt to connect to ports behind which there are no services, and th…

Scrapers and port scanners are mostly entirely separate activities. If you don't have any unexpected open ports you have nothing to fear from a port scanner, and blocking them won't stop you from getting attacked by scrapers.

Post by TZubiri » Mon, Jul 13, 2026, 4:37 PM UTC

CodesInChaos wrote: post #238

TZubiri wrote: post #77

>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on. The first argu…

> The moment it does you'll be able to tell because Bitcoin will crash to 0. The "workaround" for PoW is running the PoW computation on hardware that's better suited for the task. Bitcoin mining has been using ASIC for many years now. Let's say a legitimate user is willing to wait for one minute on a budget phone. Then your PoW is limited to what that phone can compute in one minute. But on the attacker's specialized…