Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LWN is currently under the heaviest scraper attack seen yet

social.kernel.org

Post reply on HN
Print view View original

81–90 of 134 posts

Re: LWN is currently under the heaviest scraper attack seen yet

#81

Post by iamnothere »

shantara wrote: post #68
iamnothere wrote: post #30

I am starting to think these are not just AI scrapers blindly seeking out data. All kinds of FOSS sites including low volume forums and blogs have been under this kind of persistent pressure for a while now. Given the cost involved in maintaining this kind of widespread constant scraping, the economics don’t seem to line up. Surely even big budget projects would adjust their scraping rates based on how many changes t…

>I haven’t heard of the same attacks facing (for instance) niche hobby communities. Does anyone know if those sites are facing the same scale of attacks? They are. I participate in modding communities for very niche gaming projects. All of them experienced massive DDOS attacks from AI scrappers on their websites over the past year. They are long running non-commercial projects that don’t present any business interest…

Thanks. The scale of this is just mind-boggling. Unbelievably wasteful.

Re: LWN is currently under the heaviest scraper attack seen yet

#82

Post by debo_ »

phil21 wrote: post #75
fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

I'd guess some sort of middle management local maxima. Someone set some metric of X pages per day scraped, or Y bits per month - whatever. CEO gets what he wants. Then that got passed down to the engineers and those engineers got ridden until they turned the dial to 11. Some VP then gets to go to the quarterly review with a "we beat our data ingestion metrics by 15%!". So any engineer that pushes back basically gets…

Why is it in these invented HN scenarios that the engineers just happen to have absolutely no agency?

Re: LWN is currently under the heaviest scraper attack seen yet

#84

Post by fancyfredbot »

overfeed wrote: post #71
fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

> If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to... You are incorrectly assuming competency, thoughtful engineering and/or some modicum of care for negative externalities. The scraper may have been whipped up by AI, and shipped an hour later after a quick 15-minute test against en.wikipedia.org. Whoever the perpetrator is, they are hiding behind "residential IP providers…

Yes I agree it's more likely incompetence than malice. That's another reason I don't think it's a lab. Even if you don't like the big labs you can probably admit they are reasonably smart/competent.

Residential IP providers definitely don't remove reputational risk. There are many ways people can find out what you are doing. The main one being that your employees might decide to tell on you.

The IP providers are a great way of getting around cloud flare etc. They are also reasonably expensive! I find it very plausible that these IP providers are involved but I still don't understand who is paying them.

Re: LWN is currently under the heaviest scraper attack seen yet

#85

Post by antod »

sandworm101 wrote: post #61
fancyfredbot wrote: post #27

Earlier quoted context omitted.

If you can bore an LLM that's exciting.

I would rather setup a "shadow" site designed only for LLMs. I would stuff it with ao much insanity that Grok would not be able to leave. How about a billion blog post where every use of "American" is replaced with "Canadian". By the time im done, grok will be spouting conspiracy theories about the decline of the strategic bacon reserve.

> By the time im done, grok will be spouting conspiracy theories about the decline of the strategic bacon reserve.

grok will blame the zionists rather than the freemasons for that one.

Re: LWN is currently under the heaviest scraper attack seen yet

#86

Post by oblio »

debo_ wrote: post #82
phil21 wrote: post #75

Earlier quoted context omitted.

I'd guess some sort of middle management local maxima. Someone set some metric of X pages per day scraped, or Y bits per month - whatever. CEO gets what he wants. Then that got passed down to the engineers and those engineers got ridden until they turned the dial to 11. Some VP then gets to go to the quarterly review with a "we beat our data ingestion metrics by 15%!". So any engineer that pushes back basically gets…

Why is it in these invented HN scenarios that the engineers just happen to have absolutely no agency?

Because: who would refuse more money?

Re: LWN is currently under the heaviest scraper attack seen yet

#87

Post by antod »

chrisjj wrote: post #67
fabian2k wrote: post #16

Earlier quoted context omitted.

Sufficiently aggressive and inconsiderate scraping is indistinguishable from a DDOS attack.

No scraper seeks to deny the service it needs. And no responsible site operator unable to distinguish should claim DDOS.

No well behaved scraper at least.

Re: LWN is currently under the heaviest scraper attack seen yet

#88

Post by rfmoz »

delfinom wrote: post #76
fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

As someone that runs the infrastructure for a large OSS project. Mostly Chinese AI firms. All the big name brand AI firms play reasonably nice and respect robots.txt. The Chinese ones are hyper aggressive, with no rate limit and pure greed scraping. They'll scrape the same content hundreds of times the same day

Chinese AI is doing large amounts of request in the past weeks.

Re: LWN is currently under the heaviest scraper attack seen yet

#89

Post by jacobgkau »

fancyfredbot wrote: post #84
overfeed wrote: post #71

Earlier quoted context omitted.

> If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to... You are incorrectly assuming competency, thoughtful engineering and/or some modicum of care for negative externalities. The scraper may have been whipped up by AI, and shipped an hour later after a quick 15-minute test against en.wikipedia.org. Whoever the perpetrator is, they are hiding behind "residential IP providers…

Yes I agree it's more likely incompetence than malice. That's another reason I don't think it's a lab. Even if you don't like the big labs you can probably admit they are reasonably smart/competent. Residential IP providers definitely don't remove reputational risk. There are many ways people can find out what you are doing. The main one being that your employees might decide to tell on you. The IP providers are a gr…

This is just an anecdote, but having been dealing with similar problems on one of my websites for the past year or so, I was experiencing a huge number of hits from different residential IP addresses (mostly Latin American) at the same time once every 5-10 minutes (which started crashing my site regularly). Digging through my server's logs and watching them in real-time, I noticed one or two Huawei IP's making requests at the same time as the dozens or hundreds of residential IP's. Blocking the Huawei IP's seemed to mysteriously cut back the residential IP requests, at least for a short amount of time (i.e. a couple of hours).

This isn't to say every attack that looks similar is being done by Huawei (which I can't say for certain, anyway). But to me, it does look an awful lot like even large organizations you'd think would be competent can stoop to these levels. I don't have an answer for you as to why.

Re: LWN is currently under the heaviest scraper attack seen yet

#90

Post by sgc »

Can somebody tell me what is a normal "cost of doing business" level of bot traffic these days? I have way too much bot traffic like everybody else, but I don't know if I am an outlier or just run of the mill. I get about 100k bot hits a day, presumably because I have about 350k pages on my site.
Post reply on HN