Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LWN is currently under the heaviest scraper attack seen yet

social.kernel.org

Post reply on HN
Print view View original

91–100 of 134 posts

Re: LWN is currently under the heaviest scraper attack seen yet

#91

Post by fancyfredbot »

phil21 wrote: post #75
fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

I'd guess some sort of middle management local maxima. Someone set some metric of X pages per day scraped, or Y bits per month - whatever. CEO gets what he wants. Then that got passed down to the engineers and those engineers got ridden until they turned the dial to 11. Some VP then gets to go to the quarterly review with a "we beat our data ingestion metrics by 15%!". So any engineer that pushes back basically gets…

I don't think this evil boss and downtrodden engineer situation can explain what we're seeing.

Your theoretical engineers would figure out pretty quickly that crashing a server slows you down and the only way to keep the boss happy is to avoid the DDOS.

Re: LWN is currently under the heaviest scraper attack seen yet

#92

Post by kay_o »

sgc wrote: post #90

Can somebody tell me what is a normal "cost of doing business" level of bot traffic these days? I have way too much bot traffic like everybody else, but I don't know if I am an outlier or just run of the mill. I get about 100k bot hits a day, presumably because I have about 350k pages on my site.

Esports vertical: I get about 5-20b bot hits per day (unwanted; includes both IA, brute forcer, "security" scanners, wp-admin/ requests), 1.5m google spider (search; respectful of crawl delay), and about 50-100m human (largely mobile).

For unwanted bots I serve incorrect information -- it's online gaming match history without much text so requests flagged as unwanted bots will, instead of heavy database queries, get plausibly random numbers -- seeded by the user so they stay stable -- KDA, win/loss rates, rankings.

A few dozen million distinct pages but they are numeric stats for user profiles, match stats with little to none paragraph form of text.

Re: LWN is currently under the heaviest scraper attack seen yet

#93

Post by hackingonempty »

iamnothere wrote: post #30

I am starting to think these are not just AI scrapers blindly seeking out data. All kinds of FOSS sites including low volume forums and blogs have been under this kind of persistent pressure for a while now. Given the cost involved in maintaining this kind of widespread constant scraping, the economics don’t seem to line up. Surely even big budget projects would adjust their scraping rates based on how many changes t…

A couple of forums I have lurked on for years have closed up and now require a login to read.

Re: LWN is currently under the heaviest scraper attack seen yet

#94

Post by fancyfredbot »

slicerdicer2 wrote: post #78
tux3 wrote: post #60

Earlier quoted context omitted.

Some guy doesn't show up with 10k residential IPs. This is deliberate and organized.

There are multiple israeli companies who will provide you with millions of residential proxies at a per gb usage rate and a very easy API. You can set this up in minutes with claude code.

These IP providers aren't cheap (cost per GB seems to be $4 but there are bulk discounts). The cost to grab all of LWN isn't prohibitively high for an individual but it's enough that most people probably wouldn't do it on a whim.

I suppose it only needs one person though. So it's probably a pretty plausible explanation.

Re: LWN is currently under the heaviest scraper attack seen yet

#95

Post by quectophoton »

wongarsu wrote: post #39
blakesterz wrote: post #13

"It is a DDOS attack involving tens of thousands of addresses" It is amazing just how distributed some of these things are. Even on the small sites that I help host we see these types of attacks from very large numbers of diverse IPs. I'd love to know how these are being run.

There are plenty of providers selling "residential proxies", distributing your crawler traffic through thousands of residential IPs. BrightData is probably the biggest, but its a big and growing market. And if you don't care about the "residential" part you can get proxies with data center IPs for much cheaper from the same providers. But those are easily blocked

And how do you get those residential IP addresses?

Well, you just need people to install your browser extension. Or your proprietary web browser. Or your mobile app. Or your nice MCP. Maybe get them to add your PPA repository so they automatically install your sneakily-overriden package the next time they upgrade their system.

Anything goes as long as your software has access to outgoing TCP port 443, which almost nobody blocks, so even if it's being run from within a Docker container or a VM it probably doesn't affect you.

Re: LWN is currently under the heaviest scraper attack seen yet

#96

Post by jacquesm »

palmotea wrote: post #48
jacquesm wrote: post #12

AI allows companies to resell open source code as if they wrote it themselves doing an end run around all license terms. This is a major problem. Of course they're not going to stop at just code. They need all the rest of it as well.

> AI allows companies to resell open source code as if they wrote it themselves doing an end run around all license terms. This is a major problem. Has it been adjudicated that AI use actually allows that? That's definitely what the AI bros want (and will loudly assert), but that doesn't mean it's true.

You are misinterpreting my use of the word 'allow'. Think of it as 'enables' or 'makes it possible'. It is fairly obvious that AI does not grant permission as well as that there was no reference to any legal proceedings.

Re: LWN is currently under the heaviest scraper attack seen yet

#97

Post by redwall_hp »

palmotea wrote: post #48
jacquesm wrote: post #12

AI allows companies to resell open source code as if they wrote it themselves doing an end run around all license terms. This is a major problem. Of course they're not going to stop at just code. They need all the rest of it as well.

> AI allows companies to resell open source code as if they wrote it themselves doing an end run around all license terms. This is a major problem. Has it been adjudicated that AI use actually allows that? That's definitely what the AI bros want (and will loudly assert), but that doesn't mean it's true.

We're just seeing that the copyright emperor has no clothes: companies want to infringe upon others' rights through a copyright laundering machine, but of course will insist that the laundered code they use is their property, protected by the holy copyright cudgel.

Re: LWN is currently under the heaviest scraper attack seen yet

#98

Post by redwall_hp »

gruez wrote: post #23
zipy124 wrote: post #14

Earlier quoted context omitted.

From the creators of easy money laundering (crypto bros), we now bring you easy money laundering 2: intellectual property laundering, coming to a theatre near you soon!

>From the creators of easy money laundering (crypto bros), Is there even any evidence that "crypto bros" and "AI bros" are even the same set of people other than being vaguely "tech" and hated by HN? At best you have someone like Altman who founded openai and had a crypto project (worldcoin), but the latter was approximately used by nobody. What about everyone else? Did Ilya Sutskever have a shitcoin a few years ago?…

Elmo Musk was pumping and dumping dogecoin for years before pivoting to starting an LLM Ponzi.

Re: LWN is currently under the heaviest scraper attack seen yet

#99

Post by Havoc »

TurdF3rguson wrote: post #77
Havoc wrote: post #70

That makes no sense. There is no reason for AI scrappers to use tens of thousands of IPs to scrape one site over and over. That just sounds like a classic DDOS.

Sure there is, scrapers do that to defeat throttling. 10,000 is less than 3 hours of scraping at 1 request per second.

It's not 10k requests, it's 10k IPs

Having lots of IPs is helpful for scraping, but you don't need 10k. That's a botnet

Re: LWN is currently under the heaviest scraper attack seen yet

#100

Post by TurdF3rguson »

Havoc wrote: post #99
TurdF3rguson wrote: post #77

Earlier quoted context omitted.

Sure there is, scrapers do that to defeat throttling. 10,000 is less than 3 hours of scraping at 1 request per second.

It's not 10k requests, it's 10k IPs Having lots of IPs is helpful for scraping, but you don't need 10k. That's a botnet

The way it works is this: You can sign up for a proxy rotator service that works like a regular proxy except every request you make goes through a different ip address. Is that a botnet? Yes. Is it also typically used in a scraping project? Yes.
Post reply on HN