And people complain about cloudflare/anubis/etc. Unfortunately, it's looking like this is the alternative.
Gentoo bugzilla closed due AI bot scraper overload
41–50 of 104 posts
Re: Gentoo bugzilla closed due AI bot scraper overload
#42Every website that wants to, can just charge for dumps. Anubis or even Cloudflare can handle the rest I think.
Scrapers will ignore the dumps. You can link to them in 10 million HTTP 429 responses, they will still ignore them.
Those scrapers can be blocked. Some scrapers won't ignore the responses, and maybe it'll lead to a meaningful reduction in scraping traffic.
Re: Gentoo bugzilla closed due AI bot scraper overload
#43Earlier quoted context omitted.
and do you happen to know a zero friction payment system that works internationally like the Internet itself does? because I promise you that unless you already have a captive audience, even a minuscule amount of friction to access your service will cost you an overwhelming percentage of organic human visitors.
Something like Chaum's blind-signature based ecash, or GNU Taler would be ideal in terms of efficiency, but it's still centralized in distribution. Freenet currently uses something like this. Another option I was thinking of would be a pretty inflationary (or demurrage) cryptocurrency in which you have some sort of RandomX or other CPU-bound PoW. A web server could act as a mining pool and use mining shares interchan…
Lightning - a fast, instant, bitcoin layer 2 network - is perfectly sufficient for micropayments. Volatility is a no-issue in this case, as you can freely trade the 5 cents in realtime into other assets and minimize holding time of BTC. You will loose the spread, ofc.
Re: Gentoo bugzilla closed due AI bot scraper overload
#44It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.
Perhaps the biggest problem is 5 cents or less isn't enough to cover minimum transaction fees.
Re: Gentoo bugzilla closed due AI bot scraper overload
#45What are they scraping the gentoo bugzilla for? I'm confused. Unless you're actively using Gentoo why would this be a resource? Very confusing. Also you'd think we'd have LLM BitTorrent by now, where if they want to scrape something we get a DHT hash for the content and share it with one another, rather than melt servers with the millionth request of the day.
They're scraping everything. It doesn't matter what. It doesn't matter if it makes sense. They just scrape it all.
Re: Gentoo bugzilla closed due AI bot scraper overload
#46It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.
[deleted]
Re: Gentoo bugzilla closed due AI bot scraper overload
#47There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable. We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time. Most scrapers are relatively honest in some way shape or form.
How did you implement this? My site's getting hammered, any tips would be appreciated
What kind of uptime/expectations/etc. are you working with?
Scraper "attacks" don't take down our robot-specific server very often; it's safe for us to take heavy-handed approaches that sometimes redirect users there. 99% (made-up high number) of the time, the misdirected users don't realize anything is amiss.
Start by analyzing your traffic, specifically user agents. Look for "robot" or even "bot" in the user agent and load balance those to a robot-specific server. This can all be done within Cloudflare. The only code is the user agent condition. Note: I'm very open to input here if anyone reading notices that we're shooting ourselves in the feet. Based on our analysis, the remaining traffic is a good picture of our human users.
We have loads of other conditions, mostly balancing specific IP ranges for entities when we know exactly who they are, but this is a good start.
Re: Gentoo bugzilla closed due AI bot scraper overload
#48Corrected title: "Gentoo bugzilla closed because the guy running it, who says it is 'unusable anyway', saw a lot of traffic from different IP addresses with no clear pattern and accuses AI"
Re: Gentoo bugzilla closed due AI bot scraper overload
#49going on a bit of a tangent here - some discussion there seems to be imply that AI bots mostly use IPv4s, which makes sense to me given that they are probably bots hosted by some cloud. Whereas IPv6 may be more organic traffic from e.g. mobile users (not in this case probably)... which made me think if at some point IPv6 may at some point win against IPv4 just because its more organic traffic (i.e. not from big tech…
My residential ISP doesn't support IPv6 at all. If a site is ipv6 only, I can't reach it unless I use a VPN.
Re: Gentoo bugzilla closed due AI bot scraper overload
#50The year is 2026, somehow peoples basic web apps are not able to keep up with scrapers. Scrapers are not new. My side load is like .01 even with 10x the traffic of last year. It's called "serving static content", "caching" and many other things that are not new concepts. Then you've got good old cloudflare which is free to use
A bugtracker is inherently dynamic.