Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

191–200 of 422 posts

Re: An update on residential proxies and the scraper situation

#191

Post by Loic »

ArtTimeInvestor wrote: post #179
Loic wrote: post #162

Earlier quoted context omitted.

Proof of work, even "custom", where the user does not need a particular interaction with the page, does not work. The scrapers are running headless Chrome and solving the work. They do not care, they do not pay the bill, the compromised system's owner pays the bill. I have such system for the registration form on one of my website to prevent the double validation of emails to be used to spam emails of victims. The Po…

Thats why I said proof of work that is used to mine a cryptocurrency to pay the bills of websites that serve information. As long as the website gets paid more than the cost of serving the pages, it does not matter if a human or a bot did the POW. Securing signup forms is another issue. Maybe related. But not what I was referring to.

Very good point, I missed the point that the PoW is paying the publisher. I will have to dig into this, this is a pretty nice idea.

Re: An update on residential proxies and the scraper situation

#192

Post by harshreality »

Chu4eeno wrote: post #143
Groxx wrote: post #79

Earlier quoted context omitted.

Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...

Except when it throws you into a reload loop. It's pretty buggy, and trivial to bypass. And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well. When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.

Putting aside the question of whether it will continue to work, even somewhat, against botnets, I find your first paragraph confusing.

Reload loops, or being able to "bypass" anubis (unless you merely mean bypassing it for the token validity period by solving a challenge), sound like misconfigurations. There's no reason for anubis itself to cause reload loops; it's tricky to configure a webserver to use it in some scenarios.

Any ability to bypass anubis probably means the site is using it in auth/challenge mode only, and then misconfigured their webserver's auth checking. Or it's a bug. If you mean the double-spend tavis mentioned in his blog post which previously made the HN frontpage, that was patched right after it was reported to the maintainer almost a year ago.

Re: An update on residential proxies and the scraper situation

#193

Post by [deleted] »

swinglock wrote: post #188
geocar wrote: post #183

Earlier quoted context omitted.

> but PoW scales Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest. > becoming much more obvious and easy to block, or they have to use massive amounts of compute. If you believe this, please contact me: I think compute is free[1] and can probably help you out. [1]: https://news.ycombinator.co…

Can you not design a PoW that is most efficient in a browser? Don't brute force hashes like Hashcash/Bitcoin, do something similar to RandomX instead but in JS. Browsers ought to run the fastest JS interpreters already so if interpreting JS becomes the bulk of the work, that attack might not work. Maybe even involve the DOM or whatever else makes sense.

[deleted]

Re: An update on residential proxies and the scraper situation

#194

Post by InsideOutSanta »

klamann wrote: post #185

I think this Anubis project is a terrible solution to the problem posed by aggressive web scrapers. Using a web browser with reasonable privacy settings has become a big loss in quality of life already, but the first time I encountered Anubis I got completely locked out of most web servers that deployed it. The situation has improved a little, but I hate that maintainers of great web services have rationalized themse…

I think all of these mitigations are unfortunate. They hurt one of the things that makes the web cool: it's a stable, stateless, idempotent way to access data.

This makes it a prime target for aggressive scraping by LLM companies, but it also makes it accessible and fast, and a prime target for benign use (like archive.org or "read later" services).

For my own sites, I'll eat the cost of the crawlers (mitigated by making the sites as efficient as possible) and keep them available to everyone.

Re: An update on residential proxies and the scraper situation

#195

Post by Jabbles »

procaryote wrote: post #186
andai wrote: post #45

Earlier quoted context omitted.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour Millions per hour is tens per second though; perhaps the fix is performance improvements

1,000,000 / 3,600 = ~278

Re: An update on residential proxies and the scraper situation

#196

Post by m00dy »

Disclaimer: I own and operate proxybase.xyz [0]

Hi HN,

I wanted to jump in and share a few thoughts. Not all the residential networks mentioned on this page are bad actors.

Transparency & Auditing: Our clients are completely open-source, and we run strict internal audits before every single release [1][2]. You don't need to be a security wizard to verify this, either. You can easily audit the code yourself—just clone the repo, feed it into an AI, and ask the right questions.

Ethical Sourcing: Consent is everything. At Proxybase, we always get explicit consent from our providers before adding them to the pool. This is exactly how ethical sourcing should be done. Historically, this industry has been incredibly shady think malware bundled into iOS/Android apps or second-tier smart TVs secretly installing background scrapers. Fortunately, the sector is finally becoming more ethically aware.

Fair Payouts: A lot of networks hold onto provider funds for months, staking them to earn passive income while making users wait. Between sky-high payout thresholds and endless waiting periods, it’s a broken system. At Proxybase, we have a $1 minimum payout sent directly to your wallet using US stablecoins.

If you have a better idea on how we can make this industry better, just lmk. I'm reading/writing on HN everyday.

[0] https://proxybase.xyz

[1] https://github.com/proxybasehq/proxybase-gui

[2] https://github.com/proxybasehq/proxybase-cli

Re: An update on residential proxies and the scraper situation

#198

Post by dorgo »

geocar wrote: post #183
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

> but PoW scales Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest. > becoming much more obvious and easy to block, or they have to use massive amounts of compute. If you believe this, please contact me: I think compute is free[1] and can probably help you out. [1]: https://news.ycombinator.co…

[flagged]

Re: An update on residential proxies and the scraper situation

#200

Post by freehorse »

miyuru wrote: post #182
Groxx wrote: post #79

Earlier quoted context omitted.

Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...

For me Cloudflare is worse, it takes more than 5 seconds, where as anubius take 1-2 secs. funny with all the IP information they have, cloudflare cannot do a better job. (I am on IPv6) and most of the time, its on marketing product pages like in framework main site, which can be cached.

Imo the worst is recaptcha. At least with cloudflare the work you have to provide is minimal. With recaptcha it can take me much longer than 5 seconds, and lately I have trouble even completing their challenges correctly. Nowadays if I see a (recaptcha) captcha I drop the site unless I must visit it for some reason, it is not worth the time, the effort or the annoyance.
Post reply on HN