Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

171–180 of 422 posts

Re: An update on residential proxies and the scraper situation

#171

Post by stavros »

andai wrote: post #45
mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

It's not AI companies scraping these websites, it's AI companies creating a massively profitable need for data, and every random Joe with vibe coded scrapers tries to make a buck out of it.

Re: An update on residential proxies and the scraper situation

#172

Post by [deleted] »

charcircuit wrote: post #109

>types of operator running residential-proxy networks to attack web sites. This is such a malicious interpretation. Do you think VPN operating are also trying to attack websites? Both offer the same kind of product. >paid for hijacking their users' network connections Nothing is being hijacked. Again the author is using wording to try and paint these people as malicious actors. >Recently, LWN was subjected what was,…

[deleted]

Re: An update on residential proxies and the scraper situation

#173

Post by coredev_ »

ArtTimeInvestor wrote: post #153

In theory, proof of work that is used to mine a cryptocurrency could be a solution. Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads. The question is which resources typical consumer hardwar…

They discuss why this might be a bad idea in the article.

Re: An update on residential proxies and the scraper situation

#174

Post by ccgreg »

mips_avatar wrote: post #144
ccgreg wrote: post #136

Earlier quoted context omitted.

If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from the AWS Open Dataset Sponsorship Program. And there's no charge for downloading it.

Thanks for making common crawl as good as it is. It’s a really important part of making the Internet better

Appreciate your kind words! Many people have worked at Common Crawl over the years, and it's been a labor of love fueled by positive comments like yours and the large list of PhD theses helped by our public web dataset.

Re: An update on residential proxies and the scraper situation

#175

Post by ccgreg »

charcircuit wrote: post #113
ccgreg wrote: post #99

Earlier quoted context omitted.

Common Crawl's archive has metadata that says when each record (html file) was crawled.

But who stores the metadata for the last date the site updated so you know if it needs to be refetched or not.

We do. First off we have a public parquet-format index of all of the urls we crawl every month. And then that also lives in a HDFS table that determines when we want to recrawl a page we've crawled before.

Re: An update on residential proxies and the scraper situation

#176

Post by armchairhacker »

harshreality wrote: post #61

> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when…

PoW can theoretically scale effectively infinite because it can mine cryptocurrency. Millions of compromised IoT devices hitting your server? Now you have enough money for a faster server.

It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.

Re: An update on residential proxies and the scraper situation

#177

Post by armchairhacker »

andai wrote: post #45
mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

It should really just be called DDoS, at a certain level of incompetence intent doesn’t matter. You’re right that there’s zero (information gathering) benefit over reasonable scraping which wouldn’t cripple the site.

Who’s doing it, are they even using the data?

Re: An update on residential proxies and the scraper situation

#178

Post by aorth »

TurdF3rguson wrote: post #51
cyanydeez wrote: post #2

mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS. The question is more about why the US and others can't properly enforce the bullshit all this amounts to.

What exactly should be illegal here? Scraping websites? AI agents? Not following robots.txt?

The excessive scraping and ignoring robots.txt only breaks the informal social contract established over the past decades of the open internet.

The real problem is the companies offering money to developers if they include unrelated SDKs in their calculator or flashlight (for example) applications. Those SDKs add functionality to incorporate those devices into a network that can be used for scraping. The traffic is little, but is distributed over millions of residential devices all over the world, making it difficult to categorize or block. That should be illegal, and that's what Google et al can be expected to be policing on their app stores.

Re: An update on residential proxies and the scraper situation

#179

Post by ArtTimeInvestor »

Loic wrote: post #162
ArtTimeInvestor wrote: post #153

In theory, proof of work that is used to mine a cryptocurrency could be a solution. Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads. The question is which resources typical consumer hardwar…

Proof of work, even "custom", where the user does not need a particular interaction with the page, does not work. The scrapers are running headless Chrome and solving the work. They do not care, they do not pay the bill, the compromised system's owner pays the bill. I have such system for the registration form on one of my website to prevent the double validation of emails to be used to spam emails of victims. The Po…

Thats why I said proof of work that is used to mine a cryptocurrency to pay the bills of websites that serve information.

As long as the website gets paid more than the cost of serving the pages, it does not matter if a human or a bot did the POW.

Securing signup forms is another issue. Maybe related. But not what I was referring to.

Re: An update on residential proxies and the scraper situation

#180

Post by armchairhacker »

coredev_ wrote: post #173
ArtTimeInvestor wrote: post #153

In theory, proof of work that is used to mine a cryptocurrency could be a solution. Bitcoin and others are already secured via massive pow computations. If we could shift that into browsers, no additional energy would be used and we could solve an issue that has been unsolved for too long: How to pay websites that provide useful information other than with ads. The question is which resources typical consumer hardwar…

They discuss why this might be a bad idea in the article.

They discuss why proof of work is bad, not crypto

> partly because it causes annoying delays for those trying to get to the site

This is true but usually a small issue. It’s further alleviated by cached tokens so you only have to solve the challenge once in a while per site, and a login token may let you skip it.

> partly because it seems inevitable that the scrapers will eventually find their way around it…A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.

Solved by making money off it.

Post reply on HN