Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

371–380 of 422 posts

Re: An update on residential proxies and the scraper situation

#371

Post by inigyou »

throwaway7356 wrote: post #231
charcircuit wrote: post #156

Earlier quoted context omitted.

>sell access to the user's internal residential network That is not what the SDK was doing. The actual code in the SDK protects against this (simplified to take less space): if (addr.isSiteLocalAddress() || addr.isLoopbackAddress()) { LogUtils.e("PopaTunnelAsyncThread", "Hacking? The Host Resolved Ip is " + addr + " on tunnel id:" + tunnelId); throw new IllegalArgumentException("Hacking? The tunnel host resolved ip i…

It'll still connect to IPv6 addresses and bypass any firewalls. Also users might become part (victim?) of a police investigation because of illegal actions that seem to originate from their local residential connection. So still good to take down such backdoors. Would be nice to go after the botnet operators as well...

Any webpage you visit can trigger an illegal action on your internet connection with a simple image tag, for example https://www.google.com/search?q=I+just+killed+my+wife.+How+s...">. It doesn't seem to be a real concern.

Re: An update on residential proxies and the scraper situation

#372

Post by colinsane »

klamann wrote: post #185

I think this Anubis project is a terrible solution to the problem posed by aggressive web scrapers. Using a web browser with reasonable privacy settings has become a big loss in quality of life already, but the first time I encountered Anubis I got completely locked out of most web servers that deployed it. The situation has improved a little, but I hate that maintainers of great web services have rationalized themse…

the options available to small web hosts are:

1. do nothing: everyone gets page loads of 30s if not worse.

2. put the content behind anubis -> load drops by 80% and page loads are back to "a couple of seconds" (including the client-side challenge).

3. take the content offline -> load drops to 0.

which of these would you prefer?

Re: An update on residential proxies and the scraper situation

#373

Post by alightsoul »

crote wrote: post #349
jazzyjackson wrote: post #190

Earlier quoted context omitted.

Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.

How is that not a massive GDPR violation?

They can just not do it in Europe and keep doing it everywhere else

Re: An update on residential proxies and the scraper situation

#374

Post by alightsoul »

fc417fc802 wrote: post #323
alightsoul wrote: post #249

Earlier quoted context omitted.

There's lists of data center IPs. You're probably in them and That's why you're getting banned

Regardless of the precise logic it's no excuse for the policy. Simply hand out a sufficiently difficult PoW to prevent widespread abuse. I'm quite certain it isn't a generic "datacenter" list though because a given VPN exit that was working will suddenly stop. Meanwhile I have a valid cookie yet that is disregarded.

Wikipedia maintains a quite exhaustive list of them to prevent spam and they also block vpn IPs as soon as they're found.

Re: An update on residential proxies and the scraper situation

#375

Post by 1vuio0pswjnm7 »

"The startup, which markets itself as the second-largest data collection firm after Alphabet Inc.s Google, has grown significantly on demand from data-hungry AI companies."

https://www.bloomberg.com/news/articles/2026-07-10/web-scrap...

Seems like Google is going after the competition

Re: An update on residential proxies and the scraper situation

#377

Post by alightsoul »

Rebelgecko wrote: post #275
alightsoul wrote: post #106

Earlier quoted context omitted.

From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.

Cloudflare often just straight up blocks me or makes me do a captcha. IMO those are both much worse than Anubis

Sounds like it's because your IP is on a data Center IP list

Re: An update on residential proxies and the scraper situation

#378

Post by chromatin »

kordlessagain wrote: post #288
dang wrote: post #69

Earlier quoted context omitted.

It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.

I came up with the idea and helped build Grub, the distributed crawler. Looksmart bought it, ran it for a time, then sold it to Wikimedia. I reclaimed the name recently (abandoned mark) and have a new crawler now that is agentic. I use it for my own research runs, and it's not my main focus at this point, nor am I trying to get it attention. A lot of LLMs and coding agents can easily fetch content if it is needed and…

I would very much like to see a Lightning-Network based 402 standard succeed!

Re: An update on residential proxies and the scraper situation

#379

Post by dlenski »

harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Anubis appears to be a temporarily-useful stopgap that has been cargo culted into prominence and an expectation of permanent usefulness, for reasons I don't fully understand.

The cost of solving the default Anubis PoW is negligible on cloud servers, and it's even lower if you use native code rather than JavaScript to solve it, which Tavis Ormandy helpfully demonstrated last year (https://lock.cmpxchg8b.com/anubis.html). If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.

So Anubis doesn't do much to stop bots, but it makes otherwise lightweight websites (little JavaScript or interactivity) almost unusable on low-resource systems like my old phone or an old Atom-based nettop.

> when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape

This "IP-bound proof-of-work" thing is gonna kill multipath TCP and bring down IPv6 with it. Uffff.

Re: An update on residential proxies and the scraper situation

#380

Post by charcircuit »

throwaway7356 wrote: post #231
charcircuit wrote: post #156

Earlier quoted context omitted.

>sell access to the user's internal residential network That is not what the SDK was doing. The actual code in the SDK protects against this (simplified to take less space): if (addr.isSiteLocalAddress() || addr.isLoopbackAddress()) { LogUtils.e("PopaTunnelAsyncThread", "Hacking? The Host Resolved Ip is " + addr + " on tunnel id:" + tunnelId); throw new IllegalArgumentException("Hacking? The tunnel host resolved ip i…

It'll still connect to IPv6 addresses and bypass any firewalls. Also users might become part (victim?) of a police investigation because of illegal actions that seem to originate from their local residential connection. So still good to take down such backdoors. Would be nice to go after the botnet operators as well...

Instead require app developers to explicitly call out this monetization method. This is neither designed to be a backdoor, nor a botnet. These are harmful classifications which can encourage people to try and target them like malware, when they are essential tools for privacy, bypassing geoblocking, and being able to collect public information from sites. The average person is not going to be involved with a police investigation. That has a tiny probability of happen and trying to blow it up to such proportions is dangerous to the existing of this important tool.
Post reply on HN