Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on residential proxies and the scraper situation

lwn.net

Post reply on HN
Print view View original

111–120 of 422 posts

Re: An update on residential proxies and the scraper situation

#111

Post by axus »

andai wrote: post #45
mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

Maybe someone is paid per scrape, without reduction in payment for duplicates.

Maybe every web query for Linux commands in $LARGE_COUNTRY checks all the Linux websites again.

Re: An update on residential proxies and the scraper situation

#112

Post by trucks-refinish »

rao-v wrote: post #74

I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands. Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use! EDIT: Lwn.net is perhaps not a…

[dead]

Re: An update on residential proxies and the scraper situation

#113

Post by charcircuit »

ccgreg wrote: post #99
dylan604 wrote: post #88

Earlier quoted context omitted.

as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?

Common Crawl's archive has metadata that says when each record (html file) was crawled.

But who stores the metadata for the last date the site updated so you know if it needs to be refetched or not.

Re: An update on residential proxies and the scraper situation

#114

Post by bigiain »

andai wrote: post #45
mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day? Edit: the article says millions of times per hour? (!?) The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a ba…

> a badly configured scraping setup?

Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s.

Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk.

It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.

Re: An update on residential proxies and the scraper situation

#115

Post by khurs »

dang wrote: post #104
khurs wrote: post #87

Earlier quoted context omitted.

Ive been seeing a 'sorry' message occasionally when accessing older pages. Is that a side effect of whatever you are doing?

It was, but there were too many legit users getting affected, so we turned most of that off a few days ago. Are you still seeing it?

Not in last day no.

Re: An update on residential proxies and the scraper situation

#116

Post by II2II »

mips_avatar wrote: post #5

I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs. I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.

I'm sure there are those who would participate, either because they want their data to be captured by AI labs or as a form of compromise.

That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).

Re: An update on residential proxies and the scraper situation

#117

Post by duskwuff »

rao-v wrote: post #74

I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands. Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use! EDIT: Lwn.net is perhaps not a…

> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.

Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.

Re: An update on residential proxies and the scraper situation

#118

Post by jolmg »

Barbing wrote: post #100
not-a-llm wrote: post #60

Earlier quoted context omitted.

by consent they mean a dialog/EULA with careful wording was display and the user clicked ok.

And even if you spent a minute explaining the proposition to a user off the street, it still wouldn't be fair unless you laid out the drawbacks. Which leads me to a question. There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?

> There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP.

Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.

Re: An update on residential proxies and the scraper situation

#119

Post by llbbdd »

BoorishBears wrote: post #94
zuzululu wrote: post #75

Earlier quoted context omitted.

ive seen unscrupulous founders fake traction during diligence, which is my day job but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection

[flagged]

> forced lowercase

??? shift is an extra key to press

Re: An update on residential proxies and the scraper situation

#120

Post by Symbiote »

jay_kyburz wrote: post #15
jay_kyburz wrote: post #13

Earlier quoted context omitted.

I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape. You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.

How about a website header with a link to a static zip that contains the whole website in one hit. The Zip could be hosted on some big public sever. Perhaps even mirrored locally for each nation.

I have essentially this at work, but the scrapers ignore it. (Or at least many, many scrapers ignore it.)
Post reply on HN