I solved this problem for my blog by simply not being interesting.
TIL about Git Brag because of your blog. It is interesting.
LWN is currently under the heaviest scraper attack seen yet
41–50 of 134 posts
Re: LWN is currently under the heaviest scraper attack seen yet
#42So which is it? DDOS attack or "AI" scrapers?
Scrapers because DDOS implies that it's malicious rather than accidental and there's no reason to think that.
Re: LWN is currently under the heaviest scraper attack seen yet
#43Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…
[deleted]
Re: LWN is currently under the heaviest scraper attack seen yet
#44Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…
I bet some guy just told Claude Code to archive all of LWN for him on a whim.
Re: LWN is currently under the heaviest scraper attack seen yet
#45Is it still ongoing? The thread appears to be over 24 hours old and as a quick test I had no issue loading the main page (which is as snappy and responsive as expected from a low-bandwidth site like LWN).
Not at the moment. It’s subsided for now.
Re: LWN is currently under the heaviest scraper attack seen yet
#46Re: LWN is currently under the heaviest scraper attack seen yet
#47I solved this problem for my blog by simply not being interesting.
[deleted]
Re: LWN is currently under the heaviest scraper attack seen yet
#48AI allows companies to resell open source code as if they wrote it themselves doing an end run around all license terms. This is a major problem. Of course they're not going to stop at just code. They need all the rest of it as well.
> AI allows companies to resell open source code as if they wrote it themselves doing an end run around all license terms. This is a major problem.
Has it been adjudicated that AI use actually allows that? That's definitely what the AI bros want (and will loudly assert), but that doesn't mean it's true.
Re: LWN is currently under the heaviest scraper attack seen yet
#49"It is a DDOS attack involving tens of thousands of addresses" It is amazing just how distributed some of these things are. Even on the small sites that I help host we see these types of attacks from very large numbers of diverse IPs. I'd love to know how these are being run.
Call it a "Distributed Intelligence Logic Denial Of Service" (DILDOS) attack both to name it distinctly and characterize the source.
Might as well call it "Artificial Intelligence Distributed Intelligence Logic Denial Of Service" (AIDILDOS) sounds about right.
Re: LWN is currently under the heaviest scraper attack seen yet
#50"It is a DDOS attack involving tens of thousands of addresses" It is amazing just how distributed some of these things are. Even on the small sites that I help host we see these types of attacks from very large numbers of diverse IPs. I'd love to know how these are being run.
In the most charitable case it's some "AI" companies with an X/Y problem. They want training data so they vibe code some naive scraper (requests is all you need!) and don't ever think to ask if maybe there's some sort of common repository of web crawls, a CommonCrawl if you will.
They don't really need to scrape training data as CommonCrawl or other content archives would be fine for training data. They don't think/know to ask what they really want: training data.
In the least charitable interpretation it's anti-social assholes that have no concept or care about negative externalities that write awful naive scrapers.