Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gentoo bugzilla closed due AI bot scraper overload

social.treehouse.systems

Post reply on HN
Print view View original

81–90 of 114 posts

Re: Gentoo bugzilla closed due AI bot scraper overload

#81

Post by rcxdude »

shevy-java wrote: post #10
mrweasel wrote: post #3

While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome. Our largest offenders seems to be mostly…

Well, some websites claimed China is behind it, which could make sense (I would not know either way). At the same time, though, I kind of doubt your carte blanche here for all those companies. Why would you think none of them are responsible for the AI slop spam? > Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation Ok. So you also don't…

Well, those companies have bots that identify themselves, and you can see what they're doing. Google especially have decades of experience of designing scrapers and seem to be able to the job of scraping the whole internet without causing problems in that time.

So, it's still possible they are also running a bunch of poorly-coded scraper bots through residential proxies, but it seems a little non-obvious why they would be doing this.

(It is worth pointing out that most of this traffic seems to be dumb: it's stuff like getting lost in generated link forests of some web apps or repeatly re-querying the same endpoint on a super-high frequency. This isn't exactly going to give a good return on investment for AI training data, especially since AIUI the main race for LLM performance now is in good quality training data)

Re: Gentoo bugzilla closed due AI bot scraper overload

#82

Post by Certhas »

rufasterisco wrote: post #67

Enough with playing around the issue. The way out of this mess is not to protect with tech that works but with principles and laws. This is not a tech problem. This is about what should or should not be legal. Nor it’s a question of having time to implement solution X or Y. If someone attacks you yes, you should have better security but you also need to have legal recourse, or it will never stop. Ddos is already ille…

I agree in principle. The tech community naturally has a tendency to reach for tech solutions to societal problems. But without jurisdiction, laws are meaningless.

If you create a law that says you have to honour robots.txt files, what do you do if an IP from another country fails to do so?

Re: Gentoo bugzilla closed due AI bot scraper overload

#83

Post by Seattle3503 »

pooploop64 wrote: post #64
mrweasel wrote: post #3

While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome. Our largest offenders seems to be mostly…

One of the defining characteristics of AI turned out to be utter and profound facelessness. The scraping, the data centers, the slop content, it's like it all comes from thin air. Nobody is talking about who is really doing all this or why, because it's always just coming from... somewhere else outside of our place.

You aren't stuck in internet traffic, you are internet traffic.

Re: Gentoo bugzilla closed due AI bot scraper overload

#84

Post by arjie »

I like lesswrong's approach to this. You get 4xx if you scrape that site but greaterwrong.com serves some kind of static site that is scrape friendly. Seems fine to me.

Though, checking, I see that it requires a login but is still up. I misunderstood what 'closed' meant. This seems fine.

Re: Gentoo bugzilla closed due AI bot scraper overload

#85

Post by rcxdude »

rufasterisco wrote: post #67

Enough with playing around the issue. The way out of this mess is not to protect with tech that works but with principles and laws. This is not a tech problem. This is about what should or should not be legal. Nor it’s a question of having time to implement solution X or Y. If someone attacks you yes, you should have better security but you also need to have legal recourse, or it will never stop. Ddos is already ille…

The question is, even if you identify who is the source of the traffic, are they even in a place where you can realistically sue them? The problematic traffic generally is not the bots that identify themselves, but the ones that are using residential proxies and try to be a non-fingerprintable as possible.

Re: Gentoo bugzilla closed due AI bot scraper overload

#86

Post by sandeepkd »

mrweasel wrote: post #3

While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome. Our largest offenders seems to be mostly…

You need to account for the fact that a lot of scraping is delegated by the big players to smaller players who can take loss of reputation and can use questionable methods (residential IPs). Some of the scrapers from these AI companies have been written very poorly from performance perspective.

From what I understand the pressure has created even Google to be a lot more aggressive than what it was before. Not entirely sure but I believe google has two categories of scrapers, the regular one and a new one for AI

Re: Gentoo bugzilla closed due AI bot scraper overload

#87

Post by kangalioo »

tarpitt wrote: post #39
comboy wrote: post #28

Earlier quoted context omitted.

Not sure it works, isn't it already like that if you have the cursed javascript-only pages? It's basically proof of work to access their content (and if you are scrapping en masse you likely also need some LLM getting involved)

With micropayments, the server owner makes 1 million requests times 0.05 cents = ~$500 The status quo with those js PoW pages doesn't really benefit the server owner at all, it's wasted energy. I mean proof of work is always wasted energy, but I figure it's better to kill two birds with one stone. CoinHive was one example of this. (I think this is a correct link? https://github.com/cazala/coin-hive ). Although I thin…

$500 would amount to just 10k requests. One million requests is $50k
Post reply on HN