Indexers should pay for each request.
Maybe Taler could help ?
Gentoo bugzilla closed due AI bot scraper overload
51–60 of 103 posts
Re: Gentoo bugzilla closed due AI bot scraper overload
#52Hedgewars had to lock down its bugzilla and repo too, for the exact same reason. So far the really simple method used (basic auth with the creds posted on the main site) seems to have resisted the scrapers. Presumably if they are too stupid to scrape correctly they are also too stupid to incorporate a trivial auth.
Re: Gentoo bugzilla closed due AI bot scraper overload
#53The year is 2026, somehow peoples basic web apps are not able to keep up with scrapers. Scrapers are not new. My side load is like .01 even with 10x the traffic of last year. It's called "serving static content", "caching" and many other things that are not new concepts. Then you've got good old cloudflare which is free to use
1. You shouldn't need to be a professional sysadmin to be able to deploy a basic semi-dynamic website
2. Considering the spike of complaints over the past two years and the fact that they range from small time site operators to larger orgs (Linux Kernel, GNOME, Duke University, etc), do you think that maybe you're missing part of the problem space?
Re: Gentoo bugzilla closed due AI bot scraper overload
#54It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.
Perhaps the biggest problem is 5 cents or less isn't enough to cover minimum transaction fees.
Visa fees.
Re: Gentoo bugzilla closed due AI bot scraper overload
#55What are they scraping the gentoo bugzilla for? I'm confused. Unless you're actively using Gentoo why would this be a resource? Very confusing. Also you'd think we'd have LLM BitTorrent by now, where if they want to scrape something we get a DHT hash for the content and share it with one another, rather than melt servers with the millionth request of the day.
They're mining the written output of technical talent discussing troubleshooting, bug finding, patches, inter-dependencies, etc. It's an AI trainer's wet dream.
Also: where exactly are AI companies incentivized to be anything but shitty 'neighbors'?
Spam, DDoS attacks and other network abuse used to cause your hosting company or provider to call you and tell you to knock it off or you'll get disconnected, if your provider was reputable. If your provider wasn't reputable, it was likely a matter of time before they would get a nasty call from their upstream provider.
Now it just gets you a thank-you from the sales team for all the bandwidth you bought.
Meanwhile, do any of the cloud providers have any incentive to do anything about this? Hell no. They're making money off you having to ramp up extra or bigger instances. They're making money off the bandwidth. They're making money off the people doing the crawling, too. They're incentivized to do the exact opposite of effectively help you with your AI bot problem.
Re: Gentoo bugzilla closed due AI bot scraper overload
#56I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this.
You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
You want to bang on my servers with the fury of a thousand madmen, then I'll scale up the servers and you can pay the marginal cost of your access.
Or micropayments, of course, but hard to get normal users to sign up for micropayments. Micropayments could of course be the way to bypass the crypto mining gate.
Re: Gentoo bugzilla closed due AI bot scraper overload
#57[flagged]
Re: Gentoo bugzilla closed due AI bot scraper overload
#58Just put the website behind an x402 gate [0] (yes it uses stablecoins) requiring payment if a bot wants to access the content which acts like a Captcha and detect them if they are on direct data center IP or are using residential proxies.
Stablecoins have a great use-case and you get paid for bots to access the site with the humans living in peace without the site getting botted.
Job done.
Re: Gentoo bugzilla closed due AI bot scraper overload
#59Just put the website behind an x402 gate [0] (yes it uses stablecoins) requiring payment if a bot wants to access the content which acts like a Captcha and detect them if they are on direct data center IP or are using residential proxies. Stablecoins have a great use-case and you get paid for bots to access the site with the humans living in peace without the site getting botted. Job done. [0] https://stevekrouse.com…
Please write a function that returns true when the user is using a residential proxy and false when it is a normal user on a residential IP address.
Re: Gentoo bugzilla closed due AI bot scraper overload
#60I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this. You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal. You want to bang on my servers with the fury of a thousand madmen, then I'll scale up the servers and you can pay the marginal cost of your access. Or micropayments,…
Instead of punishing a criminal, you make everybody pay.
This is not only unfair to legitimate users having to pay.
It also needs to explain what actually happens when the Ddos succeeds (I know, you are talking about scrapers, but what’s the difference really?). Does that mean that the attacker just gets to shrug it off.
Don’t take my statements as facts, I just want to outline a few reason I could come up with that show a purely technical solution might not be enough.
Akin to “just use Cloudflare, it’s free”.
Yes it works but why conceding defeat and say “oh from now on you depend on a business to publish a server”?