Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gentoo bugzilla closed due AI bot scraper overload

social.treehouse.systems

Post reply on HN
Print view View original

11–20 of 90 posts

Re: Gentoo bugzilla closed due AI bot scraper overload

#12

Post by cogman10 »

ComputerPerson wrote: post #4

There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable. We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time. Most scrapers are relatively honest in some way shape or form.

One thing that surpised me about gentoo is just how low budget it is as an operation. They are doing everything with a $12k budget. [1]

[1] https://www.gentoo.org/news/2026/01/05/new-year.html

Re: Gentoo bugzilla closed due AI bot scraper overload

#14

Post by kangalioo »

littlecranky67 wrote: post #11

It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.

0.05 cents even. The beautiful thing is, minuscule amounts of micropayments are already enough to fix the incentives

Re: Gentoo bugzilla closed due AI bot scraper overload

#15

Post by TingPing »

neuroticnews25 wrote: post #5
c-fe wrote: post #2

going on a bit of a tangent here - some discussion there seems to be imply that AI bots mostly use IPv4s, which makes sense to me given that they are probably bots hosted by some cloud. Whereas IPv6 may be more organic traffic from e.g. mobile users (not in this case probably)... which made me think if at some point IPv6 may at some point win against IPv4 just because its more organic traffic (i.e. not from big tech…

This is weird, every VPS I've rented in the last 8 years had IPv6, and some didn't even had IPv4.

If a problematic bot is on any mainstream VPS it will just get shut down.

Re: Gentoo bugzilla closed due AI bot scraper overload

#17

Post by Symbiote »

ComputerPerson wrote: post #4

There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable. We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time. Most scrapers are relatively honest in some way shape or form.

> Most scrapers are relatively honest in some way shape or form.

Did you miss a "dis" in there?

Re: Gentoo bugzilla closed due AI bot scraper overload

#18

Post by Symbiote »

TingPing wrote: post #15
neuroticnews25 wrote: post #5

Earlier quoted context omitted.

This is weird, every VPS I've rented in the last 8 years had IPv6, and some didn't even had IPv4.

If a problematic bot is on any mainstream VPS it will just get shut down.

How would I shut down a bot on Digital Ocean making over a million requests a day?
Post reply on HN