Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LWN is currently under the heaviest scraper attack seen yet

social.kernel.org

Post reply on HN
Print view View original

121–130 of 134 posts

Re: LWN is currently under the heaviest scraper attack seen yet

#121

Post by danaris »

iamnothere wrote: post #30

I am starting to think these are not just AI scrapers blindly seeking out data. All kinds of FOSS sites including low volume forums and blogs have been under this kind of persistent pressure for a while now. Given the cost involved in maintaining this kind of widespread constant scraping, the economics don’t seem to line up. Surely even big budget projects would adjust their scraping rates based on how many changes t…

Just as an additional anecdata point:

I run a small, niche browser game (~125 weekly unique users, down from around 1500 at its peak 15 years ago), and until I put its Wiki behind a login wall a few months ago, we were getting absolutely hammered by the bots. Not open source, not anything of particular interest to anyone beyond those already playing the game and the very select group of people who, if they found it, might actually enjoy it. (It's all text, almost-entirely-player-driven, and can be very slow at times, so people used to modern mobile games and similar dopamine factories tend to bounce off of it very quickly.)

Some of the UAs we saw included Claude and OpenAI, but there were a lot of obviously-bot requests to the Wiki that were using generic UAs and residential IPs.

If there's a concerted effort to swamp open-source projects, it's not the only thing going on. I think it's much more likely that the primary cause of this flood is people who a) think they have the right to absolutely everything on the internet, b) expect everyone they scrape from to be actively trying to hide the data from them (so, for instance, they will ignore any exposed API), and c) don't care either how many resources they use, or how much damage they do.

Re: LWN is currently under the heaviest scraper attack seen yet

#122

Post by sigseg1v »

kimixa wrote: post #56
jacquesm wrote: post #12

AI allows companies to resell open source code as if they wrote it themselves doing an end run around all license terms. This is a major problem. Of course they're not going to stop at just code. They need all the rest of it as well.

I worked on an extremely niche project revolving around an old DOS game. Code I worked on is often pretty much the only reference for some things. It's trivially easy to get claude to scrape that and regurgitate it under any requested licence (some variable names changes, but exactly the same structure - though it got one of the lookup tables wrong, which is one of the few things you could argue aren't copyrighted th…

The overwhelming majority of devs do not concern themselves with nor are even familiar with the concept of software licenses, let alone how to abide by them. I argue that it's not that they think it's "free game to [...] re-license" so much so as they think it's just code and they can use it without the idea of a licence ever even crossing their mind.

Source: find literally anything on GitHub using dependencies that are MIT licensed and being distributed without following the terms that state you must also redistribute the licence for each

Re: LWN is currently under the heaviest scraper attack seen yet

#123

Post by Havoc »

TurdF3rguson wrote: post #111
Havoc wrote: post #110

Earlier quoted context omitted.

Yeah I know, I've done scrapping too. It can absolutely be that, but that requires a confluence of multiple factors - misconfigured scrapper hitting the site over and over, a big bot net like proxy setup that is way overkilled for scrapping, a setup sophisticated enough to do all that yet simultaneously stupid enough to not cope with a site is mostly text and a couple gigs at most and all that over extended timeframe…

Except that I think it's clear that the motive was getting the data not taking the site offline. The evidence for that is that it stopped on its own without them doing anything to mitigate it. Also I don't know why you think this is sophisticated, it's probably 40 lines of Python code max.

No, DDOS do stop on their own too..

This stopping is absolutely not "evidence" that the motive was grabbing data. Honestly...

Re: LWN is currently under the heaviest scraper attack seen yet

#124

Post by jacquesm »

sigseg1v wrote: post #122
kimixa wrote: post #56

Earlier quoted context omitted.

I worked on an extremely niche project revolving around an old DOS game. Code I worked on is often pretty much the only reference for some things. It's trivially easy to get claude to scrape that and regurgitate it under any requested licence (some variable names changes, but exactly the same structure - though it got one of the lookup tables wrong, which is one of the few things you could argue aren't copyrighted th…

The overwhelming majority of devs do not concern themselves with nor are even familiar with the concept of software licenses, let alone how to abide by them. I argue that it's not that they think it's "free game to [...] re-license" so much so as they think it's just code and they can use it without the idea of a licence ever even crossing their mind. Source: find literally anything on GitHub using dependencies that…

It goes much further than that. They use LLMs to create code for them and then they claim ownership to the code.

I think that is one of the main reasons there is so much pushback against this, a lot of people are now addicted to their stream of washed code and want to claim ownership over what is essentially a derived work. The key then becomes 'if a work could not have been written by the author that claims it does that claim survive'. I think it should not but there is plenty of disagreement on this.

Re: LWN is currently under the heaviest scraper attack seen yet

#125

Post by jacquesm »

Sharlin wrote: post #63
palmotea wrote: post #48

Earlier quoted context omitted.

> AI allows companies to resell open source code as if they wrote it themselves doing an end run around all license terms. This is a major problem. Has it been adjudicated that AI use actually allows that? That's definitely what the AI bros want (and will loudly assert), but that doesn't mean it's true.

I don't think so. Because LLMs aren't legal persons (yet?!), they can neither have copyright to anything nor violate someone else's copyright. IANAL but the most reasonable legal interpretation is likely that any IP violations are actually committed by whoever it was who asked an LLM to "rewrite" something in a way that obviously counts as a derived work rather than a cleanroom implementation.

Correct.

Re: LWN is currently under the heaviest scraper attack seen yet

#126

Post by sanbor »

jacquesm wrote: post #124
sigseg1v wrote: post #122

Earlier quoted context omitted.

The overwhelming majority of devs do not concern themselves with nor are even familiar with the concept of software licenses, let alone how to abide by them. I argue that it's not that they think it's "free game to [...] re-license" so much so as they think it's just code and they can use it without the idea of a licence ever even crossing their mind. Source: find literally anything on GitHub using dependencies that…

It goes much further than that. They use LLMs to create code for them and then they claim ownership to the code. I think that is one of the main reasons there is so much pushback against this, a lot of people are now addicted to their stream of washed code and want to claim ownership over what is essentially a derived work. The key then becomes 'if a work could not have been written by the author that claims it does…

Isn’t similar to looking up/copying code from stack overflow, Google or books? Use it as reference to write the code and claim ownership. My little understanding is that the whole copyright free ride for LLMS is because it is similar to the process of humans using content under copyright as reference to create something new and claim ownership.

Re: LWN is currently under the heaviest scraper attack seen yet

#128

Post by pancsta »

overfeed wrote: post #71
fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

> If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to... You are incorrectly assuming competency, thoughtful engineering and/or some modicum of care for negative externalities. The scraper may have been whipped up by AI, and shipped an hour later after a quick 15-minute test against en.wikipedia.org. Whoever the perpetrator is, they are hiding behind "residential IP providers…

“Residential IP” sounds expensive though…

Re: LWN is currently under the heaviest scraper attack seen yet

#129

Post by 2OEH8eoCRo0 »

seb1204 wrote: post #117
2OEH8eoCRo0 wrote: post #51

When are we going to start suing these assholes? Why isn't anybody leveraging the legal system? You're all searching for technical solutions to a legal problem and fighting with one hand behind your back.

Is it possible to attribute the attack to a company?

Nope. It's impossible to trace anything that happens on the internet.

Re: LWN is currently under the heaviest scraper attack seen yet

#130

Post by overfeed »

pancsta wrote: post #128
overfeed wrote: post #71

Earlier quoted context omitted.

> If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to... You are incorrectly assuming competency, thoughtful engineering and/or some modicum of care for negative externalities. The scraper may have been whipped up by AI, and shipped an hour later after a quick 15-minute test against en.wikipedia.org. Whoever the perpetrator is, they are hiding behind "residential IP providers…

“Residential IP” sounds expensive though…

Scraper proxying is another way to monetize botnets, and I suspect those "providers" are not expensive.
Post reply on HN