Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LWN is currently under the heaviest scraper attack seen yet

social.kernel.org

Post reply on HN
Print view View original

101–110 of 134 posts

Re: LWN is currently under the heaviest scraper attack seen yet

#101

Post by dzaima »

NitpickLawyer wrote: post #62
spinningslate wrote: post #54

Earlier quoted context omitted.

I’m going to presume good faith rather than trolling. Some questions for you: 1. Coding assistants have emerged as as one of the primary commercial opportunities for AI models. As GP pointed out, LWN is the primary discussion for kernel development. If you were gathering training data for a model, and coding assistance is one of your goals, and you know of a primary sources of open source development expertise, would…

I train coding models with RLVR because that's what works. There's ~0.000x good signal in mailing lists that isn't in old mailing lists. (and, since I can't reply to the other person, I mean old as in established, it is in no way a dig to lwn). You seem to be missing my point. There is 0 incentives for AI training companies to behave like this. All that data is already in the common crawls that every lab uses. This i…

Old scrapes can't have data about new things though; have to continously re-scan to not be stuck with ancient info.

some scrapers might skip out on already-scraped sources, but easy to imagine that some/many just would not bother (you don't know if it's updated until you've checked, after all). And to some extend you do have to re-scrape, if just to find links to the new stuff.

Re: LWN is currently under the heaviest scraper attack seen yet

#103

Post by suburban_strike »

fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

When faced with evidence of operating procedure for the malicious, we forever take them at their word when they insist they're just incompetent.

The spirit of this site is so dead. Where are the hackers? Scraping is the best anyone is coming up with?

It's not scraping. They'd notice themselves getting banned everywhere for abuse of this magnitude, which is counterproductive to scraping goals. Rather than rate-limit the queries to avoid that attention, they're going out of their way to (pay to?) route traffic through a residential botnet so they can sustain it. This is not by accident, nor a byproduct of sloppy code Claude shat out. Someone wants to operate with this degree of aggressiveness, and they do not want to be detected or stopped.

This setup is as close to real-time surveillance as can be. Someone really wants to know what is being published on target sites with as minimal a refresh rate as possible and zero interference. It's not a western governmental entity or they'd just tap it.

As for who...there's only one group on the planet so obsessed with monitoring and policing everything everyone else is doing.

Re: LWN is currently under the heaviest scraper attack seen yet

#104

Post by phil21 »

debo_ wrote: post #82
phil21 wrote: post #75

Earlier quoted context omitted.

I'd guess some sort of middle management local maxima. Someone set some metric of X pages per day scraped, or Y bits per month - whatever. CEO gets what he wants. Then that got passed down to the engineers and those engineers got ridden until they turned the dial to 11. Some VP then gets to go to the quarterly review with a "we beat our data ingestion metrics by 15%!". So any engineer that pushes back basically gets…

Why is it in these invented HN scenarios that the engineers just happen to have absolutely no agency?

Because I've personally seen it. Engineer says this is silly, it will blow up in the long run - told to implement it anyways. Not much to lose for the engineer to simply do it. Substitute engineer for any line level employee in any industry and it works just as well.

I've also run into these local maxima stupidities dozens or more time in my career where it was obvious someone was gaming a performance metric at the expense of the bigger picture - which required escalation to someone who could see said bigger picture to get fixed. Happens all the time as a customer where some sales rep or sales manager wants to game short-term numbers at the expense of long-term relationships. Smaller companies you can usually get it fixed pretty quickly, larger companies tend to do more doubling down.

It usually starts with generally well-intentioned goal setting but devolves into someone optimizing a number on a spreadsheet without care (or perhaps knowledge) of the damage it can cause.

Hell, for the most extreme example look at Dieselgate. Those things don't start from some evil henchman at the top saying "lets cheat and game the metrics" - it often starts with someone setting impossible to achieve goals unknowingly in service of "setting the bar high for the organization", and by the time the backpressure filters up through the org it's oftentimes too late to fix the damage.

Re: LWN is currently under the heaviest scraper attack seen yet

#105

Post by tjons »

rfmoz wrote: post #88
delfinom wrote: post #76

Earlier quoted context omitted.

As someone that runs the infrastructure for a large OSS project. Mostly Chinese AI firms. All the big name brand AI firms play reasonably nice and respect robots.txt. The Chinese ones are hyper aggressive, with no rate limit and pure greed scraping. They'll scrape the same content hundreds of times the same day

Chinese AI is doing large amounts of request in the past weeks.

how is this showing up for you? site you host or bigger scale? I'm not surprised but rather curious.

Re: LWN is currently under the heaviest scraper attack seen yet

#106

Post by tjons »

gruez wrote: post #23
zipy124 wrote: post #14

Earlier quoted context omitted.

From the creators of easy money laundering (crypto bros), we now bring you easy money laundering 2: intellectual property laundering, coming to a theatre near you soon!

>From the creators of easy money laundering (crypto bros), Is there even any evidence that "crypto bros" and "AI bros" are even the same set of people other than being vaguely "tech" and hated by HN? At best you have someone like Altman who founded openai and had a crypto project (worldcoin), but the latter was approximately used by nobody. What about everyone else? Did Ilya Sutskever have a shitcoin a few years ago?…

It's never/rarely the leaders. It's always the followers. This site has a high bar for engagement given the depth of subject matter, but if you check another social medium I promise you'll see this pattern. All over.

Re: LWN is currently under the heaviest scraper attack seen yet

#107

Post by suburban_strike »

delfinom wrote: post #76
fancyfredbot wrote: post #21

Who are these agressive scrapers run by? It is difficult to figure out the incentives here. Why would anyone want to pull data from LWN (or any other site) at a rate which would cause a DDOS like attack? If I run a big data hungry AI lab consuming training data at 100Gb/s it's much much easier to scrape 10,000 sites at 10Mb/s than DDOS a smaller number of sites with more traffic. Of course the big labs want this data…

As someone that runs the infrastructure for a large OSS project. Mostly Chinese AI firms. All the big name brand AI firms play reasonably nice and respect robots.txt. The Chinese ones are hyper aggressive, with no rate limit and pure greed scraping. They'll scrape the same content hundreds of times the same day

The Chinese are also sloppy. They will run those scrapers until they get banned and not give a fuck.

In my experience, they do not bother putting in the effort to obfuscate source or evade bans in the first place. They might try again later, but this particular setup was specifically engineered for resiliency.

Re: LWN is currently under the heaviest scraper attack seen yet

#108

Post by sgc »

kay_o wrote: post #92
sgc wrote: post #90

Can somebody tell me what is a normal "cost of doing business" level of bot traffic these days? I have way too much bot traffic like everybody else, but I don't know if I am an outlier or just run of the mill. I get about 100k bot hits a day, presumably because I have about 350k pages on my site.

Esports vertical: I get about 5-20b bot hits per day (unwanted; includes both IA, brute forcer, "security" scanners, wp-admin/ requests), 1.5m google spider (search; respectful of crawl delay), and about 50-100m human (largely mobile). For unwanted bots I serve incorrect information -- it's online gaming match history without much text so requests flagged as unwanted bots will, instead of heavy database queries, get…

Well you must be an outlier! 100-200x bot to human traffic is a lot. AI bots likely focus on stats / technical info if they happen to be tuned to discern, and there is money in sports stats; so you are a target.

Re: LWN is currently under the heaviest scraper attack seen yet

#109

Post by bediger4000 »

suburban_strike wrote: post #107
delfinom wrote: post #76

Earlier quoted context omitted.

As someone that runs the infrastructure for a large OSS project. Mostly Chinese AI firms. All the big name brand AI firms play reasonably nice and respect robots.txt. The Chinese ones are hyper aggressive, with no rate limit and pure greed scraping. They'll scrape the same content hundreds of times the same day

The Chinese are also sloppy. They will run those scrapers until they get banned and not give a fuck. In my experience, they do not bother putting in the effort to obfuscate source or evade bans in the first place. They might try again later, but this particular setup was specifically engineered for resiliency.

Is this an example of that "chabuduo" we read about now and then?

Re: LWN is currently under the heaviest scraper attack seen yet

#110

Post by Havoc »

TurdF3rguson wrote: post #100
Havoc wrote: post #99

Earlier quoted context omitted.

It's not 10k requests, it's 10k IPs Having lots of IPs is helpful for scraping, but you don't need 10k. That's a botnet

The way it works is this: You can sign up for a proxy rotator service that works like a regular proxy except every request you make goes through a different ip address. Is that a botnet? Yes. Is it also typically used in a scraping project? Yes.

Yeah I know, I've done scrapping too.

It can absolutely be that, but that requires a confluence of multiple factors - misconfigured scrapper hitting the site over and over, a big bot net like proxy setup that is way overkilled for scrapping, a setup sophisticated enough to do all that yet simultaneously stupid enough to not cope with a site is mostly text and a couple gigs at most and all that over extended timeframe without anyone realising their scrapper is stuck.

Or alternative explanation: It's a DDOS

Post reply on HN