Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

141–150 of 422 posts

Re: An update on residential proxies and the scraper situation

#141

>types of operator running residential-proxy networks to attack web sites. This is such a malicious interpretation. Do you think VPN operating are also trying to attack websites? Both offer the same kind of product. >paid for hijacking their users' network connections Nothing is being hijacked. Again the author is using wording to try and paint these people as malicious actors. >Recently, LWN was subjected what was,…

> Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.

If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network.

It doesn't seem like monopoly abuse to exclude such malware from application stores, just like key loggers or apps intercepting other apps network traffic without the user being aware of it (say the banking app's network traffic and password entry).

Re: An update on residential proxies and the scraper situation

#142

Earlier quoted context omitted.

I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck. Then again, a large portion of the problem seems to be bots making way too many requests and in general not being…

There are PoW approaches that even the playing field between data centers and desktops. RandomX is my favorite.

For what it's worth I'm working on hashx support. It's just going to take a bit to ship while I do browser testing with broken browser configs.

Re: An update on residential proxies and the scraper situation

#143
post #79

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...

Except when it throws you into a reload loop. It's pretty buggy, and trivial to bypass.

And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.

When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.

Re: An update on residential proxies and the scraper situation

#144
post #136
post #116

Earlier quoted context omitted.

I'm sure there are those who would participate, either because they want their data to be captured by AI labs or as a form of compromise. That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I…

If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from the AWS Open Dataset Sponsorship Program. And there's no charge for downloading it.

Thanks for making common crawl as good as it is. It’s a really important part of making the Internet better

Re: An update on residential proxies and the scraper situation

#145

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Or you can go full Reddit and just block anything that seems even remotely suspicious. Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too. Using VPN? Blocked. Your iPhone is too old, blocked. Your screen brightness too low? Believe or not, blocked.

> Your screen brightness too low? Believe or not, blocked.

... What?!

Re: An update on residential proxies and the scraper situation

#146

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Or you can go full Reddit and just block anything that seems even remotely suspicious. Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too. Using VPN? Blocked. Your iPhone is too old, blocked. Your screen brightness too low? Believe or not, blocked.

Even worse, not blocked, shadowbanned.

Re: An update on residential proxies and the scraper situation

#147

Earlier quoted context omitted.

Seriously, dang? 10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned. This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose acti…

> Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency. I took a look at the most recent comments from both accounts and they don't look similar to me in this respect. I think there are two questions here though: 1. Was the violation egregious? 2. Did it deserve an immediate ban, or did they deserve a warning etc.? Seems to me the answer to (1) is y…

Jon's been around a while and some of the piss and vinegar of youth may have subsided. He does tend to show up with LWN comes up, whether as a topic of discussion (or more often) from submitted articles. That's his baliwick, and again, I don't fault him at all for it.

Our other friend here is a more recent participant to HN, at least under this handle. (I don't know that there are others, only what I can see from this one.)

Re: An update on residential proxies and the scraper situation

#148
post #85

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

At least anubis works for me. (I run umatrix) Unfortunately whatever HN is using routinely blocks my login with "Sorry." some websites just always give me 403.

[dead]

Re: An update on residential proxies and the scraper situation

#149

I find the notion that you would use residential proxies to scrape LWN somewhat laughable, I'm reading this article using a VPN. residential proxy bandwidth isn't that cheap, I could see it be used on a reddit (though i would probably just mass register accounts to bypass their block instead).

I ran a gitweb server which was battered by bots so I eventually had to take it down. Gitweb! You can just connect using the git protocol and download everything vastly more efficiently!

In other words, they don't care at all. For them, residential bandwidth is completely free.

Re: An update on residential proxies and the scraper situation

#150
post #29
post #22

Earlier quoted context omitted.

A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.

Cloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.

cloudflare routinely denies my human-piloted browser now, on many sites.
Post reply on HN