Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LWN is currently under the heaviest scraper attack seen yet

social.kernel.org

Post reply on HN
Print view View original

71–80 of 134 posts

Re: LWN is currently under the heaviest scraper attack seen yet

#71

Post by overfeed »

fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

> If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to...

You are incorrectly assuming competency, thoughtful engineering and/or some modicum of care for negative externalities. The scraper may have been whipped up by AI, and shipped an hour later after a quick 15-minute test against en.wikipedia.org.

Whoever the perpetrator is, they are hiding behind "residential IP providers" so there's no reputational risks. Further, AI companies already have a reputation for engaging in distasteful practices, but popular wisdom claims that they make up for the awfulness with utility, so even if it turns out to be a big org like OpenAI or Anthropic, people will shrug their shoulders and move on.

Re: LWN is currently under the heaviest scraper attack seen yet

#72

Post by pstuart »

dannyobrien wrote: post #59
dannyobrien wrote: post #57

Earlier quoted context omitted.

I've been asking this for a while, especially as a lot of the early blame went on the big, visible US companies like OpenAI and Anthropic. While their incentives are different from search engines (as someone said early on in this onslaught, "a search engine needs your site to stay up; an AI company doesn't"), that's quite a subtle incentive difference. Just avoiding the blocks that inevitably spring up when you misbe…

(and if anyone from Anthropic or OpenAI is reading this: teach your models to be polite when they write crawlers! It's actually an interesting alignment issue that they don't consider the externalities of their actions right now!)

Hell, they should at least be caching those requests rather than hitting the endpoint on every single AI request that needs the info.

Re: LWN is currently under the heaviest scraper attack seen yet

#73

Post by pkaeding »

gruez wrote: post #23
zipy124 wrote: post #14

Earlier quoted context omitted.

From the creators of easy money laundering (crypto bros), we now bring you easy money laundering 2: intellectual property laundering, coming to a theatre near you soon!

>From the creators of easy money laundering (crypto bros), Is there even any evidence that "crypto bros" and "AI bros" are even the same set of people other than being vaguely "tech" and hated by HN? At best you have someone like Altman who founded openai and had a crypto project (worldcoin), but the latter was approximately used by nobody. What about everyone else? Did Ilya Sutskever have a shitcoin a few years ago?…

Okay, but "fad-use-of-GPU bros" doesn't roll off the tongue as well.

Re: LWN is currently under the heaviest scraper attack seen yet

#74

Post by alephnerd »

fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

> If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic

A little over a decade ago (f*ck I'm old now [0]), I had a similar conversation with an ML Researcher@Nvidia. Their response was "even if we are overtraining, it's a good problem to have because we can reduce our false negative rate".

Everyone continues to have an incentive to optimize for TP and FP at the expense of FN.

[0] - https://m.youtube.com/watch?v=BGrfhsxxmdE

Re: LWN is currently under the heaviest scraper attack seen yet

#75

Post by phil21 »

fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

I'd guess some sort of middle management local maxima. Someone set some metric of X pages per day scraped, or Y bits per month - whatever. CEO gets what he wants.

Then that got passed down to the engineers and those engineers got ridden until they turned the dial to 11. Some VP then gets to go to the quarterly review with a "we beat our data ingestion metrics by 15%!".

So any engineer that pushes back basically gets told too bad, do it anyways.

Re: LWN is currently under the heaviest scraper attack seen yet

#76

Post by delfinom »

fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

As someone that runs the infrastructure for a large OSS project. Mostly Chinese AI firms. All the big name brand AI firms play reasonably nice and respect robots.txt.

The Chinese ones are hyper aggressive, with no rate limit and pure greed scraping. They'll scrape the same content hundreds of times the same day

Re: LWN is currently under the heaviest scraper attack seen yet

#77

Post by TurdF3rguson »

Havoc wrote: post #70

That makes no sense. There is no reason for AI scrappers to use tens of thousands of IPs to scrape one site over and over. That just sounds like a classic DDOS.

Sure there is, scrapers do that to defeat throttling. 10,000 is less than 3 hours of scraping at 1 request per second.

Re: LWN is currently under the heaviest scraper attack seen yet

#78

Post by slicerdicer2 »

tux3 wrote: post #60
velox_neb wrote: post #44

Earlier quoted context omitted.

I bet some guy just told Claude Code to archive all of LWN for him on a whim.

Some guy doesn't show up with 10k residential IPs. This is deliberate and organized.

There are multiple israeli companies who will provide you with millions of residential proxies at a per gb usage rate and a very easy API. You can set this up in minutes with claude code.

Re: LWN is currently under the heaviest scraper attack seen yet

#79

Post by [deleted] »

dannyobrien wrote: post #57
fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

I've been asking this for a while, especially as a lot of the early blame went on the big, visible US companies like OpenAI and Anthropic. While their incentives are different from search engines (as someone said early on in this onslaught, "a search engine needs your site to stay up; an AI company doesn't"), that's quite a subtle incentive difference. Just avoiding the blocks that inevitably spring up when you misbe…

[deleted]

Re: LWN is currently under the heaviest scraper attack seen yet

#80

Post by prussia »

themafia wrote: post #29
gruez wrote: post #23

Earlier quoted context omitted.

>From the creators of easy money laundering (crypto bros), Is there even any evidence that "crypto bros" and "AI bros" are even the same set of people other than being vaguely "tech" and hated by HN? At best you have someone like Altman who founded openai and had a crypto project (worldcoin), but the latter was approximately used by nobody. What about everyone else? Did Ilya Sutskever have a shitcoin a few years ago?…

> and had a crypto project (worldcoin) That was a biometric surveillance project disguised as a crypto project. > Is there even any evidence that "crypto bros" and "AI bros" are even the same set of people No, the "AI" people are far worse. I always had a choice to /not/ use crypto. The "AI" people want to hamfistedly shove their flawed investment into every product under the sun.

I think the worst of the crypto people are probably the worst of the AI people too. Power/money-hungry grifters naturally move on to the most profitable grift when the old one peters out.
Post reply on HN