Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LWN is currently under the heaviest scraper attack seen yet

social.kernel.org

Post reply on HN
Print view View original

111–120 of 134 posts

Re: LWN is currently under the heaviest scraper attack seen yet

#111

Post by TurdF3rguson »

Havoc wrote: post #110
TurdF3rguson wrote: post #100

Earlier quoted context omitted.

The way it works is this: You can sign up for a proxy rotator service that works like a regular proxy except every request you make goes through a different ip address. Is that a botnet? Yes. Is it also typically used in a scraping project? Yes.

Yeah I know, I've done scrapping too. It can absolutely be that, but that requires a confluence of multiple factors - misconfigured scrapper hitting the site over and over, a big bot net like proxy setup that is way overkilled for scrapping, a setup sophisticated enough to do all that yet simultaneously stupid enough to not cope with a site is mostly text and a couple gigs at most and all that over extended timeframe…

Except that I think it's clear that the motive was getting the data not taking the site offline. The evidence for that is that it stopped on its own without them doing anything to mitigate it.

Also I don't know why you think this is sophisticated, it's probably 40 lines of Python code max.

Re: LWN is currently under the heaviest scraper attack seen yet

#112

Post by ofrzeta »

fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

Recently I needed to block some scrapers to execessive load on a server, and here are some that I identified:

BOTS=( "semrushbot" "petalbot" "aliyunsecbot" "amazonbot" "claudebot" "thinkbot" "perplexitybot" "openai.com/bot" )

This was really just emergency blocking and it included more than 1500 IP addresses.

Here's Amazon's page about their bot with more information including IP addresses

https://developer.amazon.com/amazonbot

Re: LWN is currently under the heaviest scraper attack seen yet

#113

Post by CodeBytes »

TiredOfLife wrote: post #102
Havoc wrote: post #70

That makes no sense. There is no reason for AI scrappers to use tens of thousands of IPs to scrape one site over and over. That just sounds like a classic DDOS.

If you call it DDOS you can't capitalize on ai hate

It likely is AI scrapers essentially doing a DDoS. They use separate IPs (and vary the UA) to prevent blocking.

I have a site which is currently being hit (over 10k requests today) and it looks like scrapers as every URL is different. If it was a DDoS, they would target costly pages like my search not every single URL.

SQLite had the same thing: https://sqlite.org/forum/forumpost/7d3eb059f81ff694 As have a few other open source repositories. It looks like badly written crawlers trying to crawl sites as fast as possible.

Re: LWN is currently under the heaviest scraper attack seen yet

#114

Post by TurdF3rguson »

quectophoton wrote: post #95
wongarsu wrote: post #39

Earlier quoted context omitted.

There are plenty of providers selling "residential proxies", distributing your crawler traffic through thousands of residential IPs. BrightData is probably the biggest, but its a big and growing market. And if you don't care about the "residential" part you can get proxies with data center IPs for much cheaper from the same providers. But those are easily blocked

And how do you get those residential IP addresses? Well, you just need people to install your browser extension. Or your proprietary web browser. Or your mobile app. Or your nice MCP. Maybe get them to add your PPA repository so they automatically install your sneakily-overriden package the next time they upgrade their system. Anything goes as long as your software has access to outgoing TCP port 443, which almost no…

Bright Data specifically offers a sdk that app developers can use monetize free games. A lot of free games and VPN apps are using it. Check out how they market it, it's wild... - https://bright-sdk.com/

Re: LWN is currently under the heaviest scraper attack seen yet

#115

Post by TurdF3rguson »

iamnothere wrote: post #55
zomiaen wrote: post #40

Earlier quoted context omitted.

How many of these scrapers are written by AI by data-science folks who don't remotely care how often they're hitting the sites, and is data they wouldn't even think to give or ask the LLM about?

But does that explain all of the various scrapers doing the same thing across the same set of sites? And again, the sheer bandwidth and CPU time involved should eventually bother the bean counters. I did think of a couple of possibilities: - Someone has a software package or list of sites out there that people are using instead of building their own scrapers, so everyone hits the same targets with the same pattern. -…

My theory on this one is some serial wantrepreneur came up with a business plan of scraping the archive and feeding it into a LLM to identify some vague opportunity. Then they paid some Fiverr / Upwork kid in India $200 to get the data. The good news is this website and any other can mitigate these things by moving to Cloudflare and it's free.

Re: LWN is currently under the heaviest scraper attack seen yet

#116

Post by Tanoc »

hackingonempty wrote: post #93
iamnothere wrote: post #30

I am starting to think these are not just AI scrapers blindly seeking out data. All kinds of FOSS sites including low volume forums and blogs have been under this kind of persistent pressure for a while now. Given the cost involved in maintaining this kind of widespread constant scraping, the economics don’t seem to line up. Surely even big budget projects would adjust their scraping rates based on how many changes t…

A couple of forums I have lurked on for years have closed up and now require a login to read.

I've wondered for a while if simple interaction systems would be good enough to fend these things off without building up walls like logins. Things like Anubis do system checks, but I'm wondering if it would be even easier to do something like the oldschool Captchas where you just have a single interactive element that requires user input to redirect to another page. Like you hit a landing page and drag a slider or click and hold to go to the page proper, things that aren't as annoying as modern Captchas and are like a fun little interactive way to enter.

As I'm writing this I'm reminded of Flash based homepages. And it really makes it apparent that Flash would be perfect for impeding these LLM crawlers.

Re: LWN is currently under the heaviest scraper attack seen yet

#117

Post by seb1204 »

2OEH8eoCRo0 wrote: post #51

When are we going to start suing these assholes? Why isn't anybody leveraging the legal system? You're all searching for technical solutions to a legal problem and fighting with one hand behind your back.

Is it possible to attribute the attack to a company?

Re: LWN is currently under the heaviest scraper attack seen yet

#118

Post by ccgreg »

fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

One way to figure that out is to look at which companies claim to have foundation models, but no one knows what their crawler is named.

I also suspect that there are a bunch of sub-contractors involved, working for companies that don't supervise them very carefully.

Re: LWN is currently under the heaviest scraper attack seen yet

#120

Post by danaris »

fancyfredbot wrote: post #84
overfeed wrote: post #71

Earlier quoted context omitted.

> If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to... You are incorrectly assuming competency, thoughtful engineering and/or some modicum of care for negative externalities. The scraper may have been whipped up by AI, and shipped an hour later after a quick 15-minute test against en.wikipedia.org. Whoever the perpetrator is, they are hiding behind "residential IP providers…

Yes I agree it's more likely incompetence than malice. That's another reason I don't think it's a lab. Even if you don't like the big labs you can probably admit they are reasonably smart/competent. Residential IP providers definitely don't remove reputational risk. There are many ways people can find out what you are doing. The main one being that your employees might decide to tell on you. The IP providers are a gr…

Sufficiently advanced incompetence is indistinguishable from malice.

Whether or not they had malice in their hearts when they implemented the scraper bots, their impact is still very much malicious.

Post reply on HN