Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

101–110 of 422 posts

Re: An update on residential proxies and the scraper situation

#101

Post by catdog »

andai wrote: post #45
mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

Maybe they have just too much money at hand, would not surprise me, people are still investing into gen AI like there is no tomorrow. Also, for the completely criminal operations, you only have to find a way to infect and distribute your bot to, e.g. some common internet of shit device. Scaling is basically free afterwards as you don't need to ask anyone. The article also hints that those are actually the biggest problem.

Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.

Re: An update on residential proxies and the scraper situation

#102

Post by catdog »

mips_avatar wrote: post #68
andai wrote: post #45

Earlier quoted context omitted.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

I kind of wish the recent Google monopoly court ruling had forced Google to open up their index to anyone, not just Perplexity/other big players.

That's really a huge issue right now (to some extent even before the AI hype) that almost everywhere google is effectively the only entity explicitly allowed to scrape.

Re: An update on residential proxies and the scraper situation

#103

Post by solidasparagus »

Bratmon wrote: post #10

Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.

This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.

Re: An update on residential proxies and the scraper situation

#104

Post by dang »

khurs wrote: post #87
dang wrote: post #69

Earlier quoted context omitted.

It's really bad. I found myself identifying with everything Jonathan wrote in the OP - so much so that I thought of asking to compare notes on mitigation measures.

Ive been seeing a 'sorry' message occasionally when accessing older pages. Is that a side effect of whatever you are doing?

It was, but there were too many legit users getting affected, so we turned most of that off a few days ago. Are you still seeing it?

Re: An update on residential proxies and the scraper situation

#105

Post by alightsoul »

andai wrote: post #45
mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

Maybe they are aggressively scanning for updates on the page

Re: An update on residential proxies and the scraper situation

#106

Post by alightsoul »

Groxx wrote: post #79
harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

Anubis is by far the least annoying throttler I encounter. Entirely agreed, just crank it up when you get a flood, I much prefer waiting a couple seconds to interacting with custom UI for tens of seconds. I'm so glad to see that (essentially) HashCash is coming back. Now we just need it for email, like it was originally designed for...

From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.

Re: An update on residential proxies and the scraper situation

#107

Post by PlasmaPower »

harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.

Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.

Re: An update on residential proxies and the scraper situation

#108

Post by asdfsa32 »

duskwuff wrote: post #96
m463 wrote: post #85

Earlier quoted context omitted.

At least anubis works for me. (I run umatrix) Unfortunately whatever HN is using routinely blocks my login with "Sorry." some websites just always give me 403.

> Unfortunately whatever HN is using routinely blocks my login with "Sorry." I believe that's the HN application itself, not a WAF in front of it.

HN is surprisingly very very guilty of a whole lot of anti-user patterns and behaviour that other companies get regularly lamented.

Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.

Re: An update on residential proxies and the scraper situation

#109

Post by charcircuit »

>types of operator running residential-proxy networks to attack web sites.

This is such a malicious interpretation. Do you think VPN operating are also trying to attack websites? Both offer the same kind of product.

>paid for hijacking their users' network connections

Nothing is being hijacked. Again the author is using wording to try and paint these people as malicious actors.

>Recently, LWN was subjected what was, by far, the heaviest scraper attack yet.

LWN is a static site. To me it seems more expensive to use Anubis than just serve the actual page.

>will now check for NetNut-infected apps

Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.

Re: An update on residential proxies and the scraper situation

#110

Post by ronsor »

mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

Ironically in early 2023 a lot of websites went out of their way to block Common Crawl. Unsurprisingly that shifted scraping toward individual actors whereas the previous solution in research was to download CC dumps and process them.
Post reply on HN