Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gentoo bugzilla closed due AI bot scraper overload

social.treehouse.systems

Post reply on HN
Print view View original

71–80 of 103 posts

Re: Gentoo bugzilla closed due AI bot scraper overload

#71

Post by FranOntanaya »

xena wrote: post #45
jpcom wrote: post #30

What are they scraping the gentoo bugzilla for? I'm confused. Unless you're actively using Gentoo why would this be a resource? Very confusing. Also you'd think we'd have LLM BitTorrent by now, where if they want to scrape something we get a DHT hash for the content and share it with one another, rather than melt servers with the millionth request of the day.

They're scraping everything. It doesn't matter what. It doesn't matter if it makes sense. They just scrape it all.

"There Will Be Blood" but it's 2026 and Plainview's child is AI.

Re: Gentoo bugzilla closed due AI bot scraper overload

#72

Post by happosai »

vlyan wrote: post #32
littlecranky67 wrote: post #11

It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.

and do you happen to know a zero friction payment system that works internationally like the Internet itself does? because I promise you that unless you already have a captive audience, even a minuscule amount of friction to access your service will cost you an overwhelming percentage of organic human visitors.

Frictionless micropayments is THE business opportunity dangling here. I'm so disappointed that the Bitcoin bros never made it easy and fast to use Bitcoin for peer to peer micropayments.

Re audience, the audience for Gentoo bugzilla went just to zero. If even 1% of the current bugzilla users would use the micropayments, that's literally infinitely more organic human visitors than today.

Re: Gentoo bugzilla closed due AI bot scraper overload

#73

Post by joezydeco »

pooploop64 wrote: post #64
mrweasel wrote: post #3

While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome. Our largest offenders seems to be mostly…

One of the defining characteristics of AI turned out to be utter and profound facelessness. The scraping, the data centers, the slop content, it's like it all comes from thin air. Nobody is talking about who is really doing all this or why, because it's always just coming from... somewhere else outside of our place.

And then it's pushed on the end users with the same amount of explanation and reason. You must adopt this now. The FOMO is insane.

Re: Gentoo bugzilla closed due AI bot scraper overload

#74

Post by calvinmorrison »

stryan wrote: post #53
calvinmorrison wrote: post #7

The year is 2026, somehow peoples basic web apps are not able to keep up with scrapers. Scrapers are not new. My side load is like .01 even with 10x the traffic of last year. It's called "serving static content", "caching" and many other things that are not new concepts. Then you've got good old cloudflare which is free to use

1. You shouldn't need to be a professional sysadmin to be able to deploy a basic semi-dynamic website 2. Considering the spike of complaints over the past two years and the fact that they range from small time site operators to larger orgs (Linux Kernel, GNOME, Duke University, etc), do you think that maybe you're missing part of the problem space?

All these experts here in advanced computing but people don't know how to cache with webpages?

Re: Gentoo bugzilla closed due AI bot scraper overload

#75

Post by vova_hn2 »

littlecranky67 wrote: post #11

It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.

The question is what exactly are you trying to achieve. Do you want to prevent them from gathering data or do you want to save resources?

If it is the latter, then, perhaps, some p2p (torrent-like) content delivery system could help.

Re: Gentoo bugzilla closed due AI bot scraper overload

#76

Post by progval »

pmdr wrote: post #38

There's a reddit thread from 9 years ago with people complaining the site blocking crawlers and thus not being indexed by search engines. So I'm not sure what they consider "overload." At this point they should dump the db on thepiratebay.

Dumping the database won't stop the scrappers because humans running the scrappers are not even aware Gentoo exists. They're just scrapping every website their bots find.

Re: Gentoo bugzilla closed due AI bot scraper overload

#78

Post by progval »

neuroticnews25 wrote: post #69
coldbrewed wrote: post #26

Earlier quoted context omitted.

Send an email to the Digitalocean abuse email with the time ranges and origin IPs.

I did this with OVH, never got any response.

abuse@ contacts never send responses (though some send acknowledgements) because they don't want to give any details about their abuse process.

Re: Gentoo bugzilla closed due AI bot scraper overload

#79

Post by xena »

vova_hn2 wrote: post #77

Are we assuming that this is the companies themselves scraping data from training or is this "agents" acting on behalf of users? Nowadays every major chat UI (ChatGPT, Claude etc) has a "tool" that allows LLM to load web pages, so it must generate some traffic.

No, most of the load comes from armies of residential IP addresses that look like Google Chrome on the wire. The major chat UIs properly identify themselves. These waves of attacks do not.

Re: Gentoo bugzilla closed due AI bot scraper overload

#80

Post by 0x_rs »

littlecranky67 wrote: post #11

It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.

Cloudflare is working exactly on enabling that, like it or not.

https://blog.cloudflare.com/monetization-gateway/

Post reply on HN