Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LWN is currently under the heaviest scraper attack seen yet

social.kernel.org

Post reply on HN
Print view View original

31–40 of 134 posts

Re: LWN is currently under the heaviest scraper attack seen yet

#31

Post by bjackman »

fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

LWN includes archives of a bunch of mailing lists so that might be a factor. There are a LOT of web on that domain.

Re: LWN is currently under the heaviest scraper attack seen yet

#32

Post by fancyfredbot »

kylehotchkiss wrote: post #24
fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

china (alibaba and tencent)

I'm not at all sure alibaba or tencent would actually want to DDOS LWN or any other popular website.

They may face less reputational damage than say Google or OpenAI would but I expect LWN has Chinese readers who would look dimly on this sort of thing. Some of those readers probably work for Alibaba and Tencent.

I'm not necessarily saying they wouldn't do it if there was some incentive to do so but I don't see the upside for them.

Re: LWN is currently under the heaviest scraper attack seen yet

#33

Post by bogwog »

gulugawa wrote: post #6

I've had luck blocking scrapers by overwriting JavaScript methods " a.getElementsByTagName = function (...args) {//Clear page content}" One can also hide components inside Shadow DOM to make it harder to scrape. However, these methods will interfere with automated testing tools such as Playwright and Selenium. Also, search engine indexing is likely to be affected.

This is a fun idea, especially if you make those functions procedurally generate garbage to get them stuck

Re: LWN is currently under the heaviest scraper attack seen yet

#34

Post by mikkupikku »

fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

NSA, trying to force everybody onto their Cloudflare reservation.

Re: LWN is currently under the heaviest scraper attack seen yet

#35

Post by PaulDavisThe1st »

blakesterz wrote: post #13

"It is a DDOS attack involving tens of thousands of addresses" It is amazing just how distributed some of these things are. Even on the small sites that I help host we see these types of attacks from very large numbers of diverse IPs. I'd love to know how these are being run.

another reference point: we've had well over 1M unique IP addresses hit git.ardour.org as part of stupid as hell git scraping effort. 1M !!!

Re: LWN is currently under the heaviest scraper attack seen yet

#36

Post by gubicle »

fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

[dead]

Re: LWN is currently under the heaviest scraper attack seen yet

#37

Post by TurdF3rguson »

gulugawa wrote: post #6

I've had luck blocking scrapers by overwriting JavaScript methods " a.getElementsByTagName = function (...args) {//Clear page content}" One can also hide components inside Shadow DOM to make it harder to scrape. However, these methods will interfere with automated testing tools such as Playwright and Selenium. Also, search engine indexing is likely to be affected.

You think you've had luck. The truth is you have no idea of knowing if this ever had any effect at all.

Re: LWN is currently under the heaviest scraper attack seen yet

#38

Post by iamnothere »

philipkglass wrote: post #26
fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

I don't think that most of them are from big-name companies. I run a personal web site that has been periodically overwhelmed by scrapers, prompting me to update my robots.txt with more disallows. The only big AI company I recognized by name was OpenAI's GPTBot. Most of them are from small companies that I'm only hearing of for the first time when I look at their user agents in the Apache logs. Probably the shadiest…

Maybe someone is putting out public “scraper lists” that small companies or even individuals can use to find potentially useful targets, perhaps with some common scraper tool they are using? That could explain it? I am also mystified by this.

Re: LWN is currently under the heaviest scraper attack seen yet

#39

Post by wongarsu »

blakesterz wrote: post #13

"It is a DDOS attack involving tens of thousands of addresses" It is amazing just how distributed some of these things are. Even on the small sites that I help host we see these types of attacks from very large numbers of diverse IPs. I'd love to know how these are being run.

There are plenty of providers selling "residential proxies", distributing your crawler traffic through thousands of residential IPs. BrightData is probably the biggest, but its a big and growing market.

And if you don't care about the "residential" part you can get proxies with data center IPs for much cheaper from the same providers. But those are easily blocked

Re: LWN is currently under the heaviest scraper attack seen yet

#40

Post by zomiaen »

iamnothere wrote: post #30

I am starting to think these are not just AI scrapers blindly seeking out data. All kinds of FOSS sites including low volume forums and blogs have been under this kind of persistent pressure for a while now. Given the cost involved in maintaining this kind of widespread constant scraping, the economics don’t seem to line up. Surely even big budget projects would adjust their scraping rates based on how many changes t…

How many of these scrapers are written by AI by data-science folks who don't remotely care how often they're hitting the sites, and is data they wouldn't even think to give or ask the LLM about?
Post reply on HN