Live data from Hacker News

Gentoo bugzilla closed due AI bot scraper overload

social.treehouse.systems

21–30 of 104 posts

Re: Gentoo bugzilla closed due AI bot scraper overload

#21

It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.

How many pages does the average software developer visit everyday? 1.000? Price at 0.001 per load and it’ll be completely impractical for crawlers but super cheap for the average connected human on earth.

Re: Gentoo bugzilla closed due AI bot scraper overload

#22

AI skynet is winning. It is stealing time from humans, thus forcing down their activity, as can be seen here. A headline would be great if those AI companies would close down. I hold them all responsible for this.

Plot twist: the AI wins not through terminators but a kafka nightmare

Re: Gentoo bugzilla closed due AI bot scraper overload

#23

The year is 2026, somehow peoples basic web apps are not able to keep up with scrapers. Scrapers are not new. My side load is like .01 even with 10x the traffic of last year. It's called "serving static content", "caching" and many other things that are not new concepts. Then you've got good old cloudflare which is free to use

Are you volunteering to setup and host that extra complexity for the Gentoo project?

Re: Gentoo bugzilla closed due AI bot scraper overload

#24
post #3

While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome. Our largest offenders seems to be mostly…

Well, some websites claimed China is behind it, which could make sense (I would not know either way). At the same time, though, I kind of doubt your carte blanche here for all those companies. Why would you think none of them are responsible for the AI slop spam? > Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation Ok. So you also don't…

We see constant abuse from the Tencent ASN/associated ACE ASN, and I've memorized the china169 backbone asn as AS4837 because of thr nonstop crawlers splattered across their network ranges. It's not possible to ID the operator running the crawlers running from these networks but there's a clear signal of the origin of some of these entities.

Re: Gentoo bugzilla closed due AI bot scraper overload

#25
post #21

It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.

How many pages does the average software developer visit everyday? 1.000? Price at 0.001 per load and it’ll be completely impractical for crawlers but super cheap for the average connected human on earth.

That would gate the internet to whole lower income countries

Re: Gentoo bugzilla closed due AI bot scraper overload

#26

Earlier quoted context omitted.

If a problematic bot is on any mainstream VPS it will just get shut down.

How would I shut down a bot on Digital Ocean making over a million requests a day?

Send an email to the Digitalocean abuse email with the time ranges and origin IPs.

Re: Gentoo bugzilla closed due AI bot scraper overload

#27

It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.

I think it's gonna kill off the internet not just for bots but to whole class of lower income countries, especially for younger learners

Re: Gentoo bugzilla closed due AI bot scraper overload

#28

It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.

0.05 cents even. The beautiful thing is, minuscule amounts of micropayments are already enough to fix the incentives

Not sure it works, isn't it already like that if you have the cursed javascript-only pages? It's basically proof of work to access their content (and if you are scrapping en masse you likely also need some LLM getting involved)

Re: Gentoo bugzilla closed due AI bot scraper overload

#29

There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable. We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time. Most scrapers are relatively honest in some way shape or form.

How did you implement this? My site's getting hammered, any tips would be appreciated

Re: Gentoo bugzilla closed due AI bot scraper overload

#30
What are they scraping the gentoo bugzilla for? I'm confused. Unless you're actively using Gentoo why would this be a resource? Very confusing. Also you'd think we'd have LLM BitTorrent by now, where if they want to scrape something we get a DHT hash for the content and share it with one another, rather than melt servers with the millionth request of the day.
Post reply on HN