I want AI to progress to a point where it can fix these static sites who fall over due to how inefficient they are.
Gentoo bugzilla closed due AI bot scraper overload
91–100 of 107 posts
Re: Gentoo bugzilla closed due AI bot scraper overload
#92There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable. We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time. Most scrapers are relatively honest in some way shape or form.
How did you implement this? My site's getting hammered, any tips would be appreciated
[flagged]
Re: Gentoo bugzilla closed due AI bot scraper overload
#93There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable. We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time. Most scrapers are relatively honest in some way shape or form.
How did you implement this? My site's getting hammered, any tips would be appreciated
[flagged]
Re: Gentoo bugzilla closed due AI bot scraper overload
#94[flagged]
Re: Gentoo bugzilla closed due AI bot scraper overload
#95Earlier quoted context omitted.
Well, some websites claimed China is behind it, which could make sense (I would not know either way). At the same time, though, I kind of doubt your carte blanche here for all those companies. Why would you think none of them are responsible for the AI slop spam? > Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation Ok. So you also don't…
Well, those companies have bots that identify themselves, and you can see what they're doing. Google especially have decades of experience of designing scrapers and seem to be able to the job of scraping the whole internet without causing problems in that time. So, it's still possible they are also running a bunch of poorly-coded scraper bots through residential proxies, but it seems a little non-obvious why they wou…
>So, it's still possible they are also running a bunch of poorly-coded scraper bots through residential proxies, but it seems a little non-obvious why they would be doing this.
why "non obvious"? The easiest explanation is gathering training data sets, and is mostly caused by AI companies guarding their pile of essentially stolen IP (given how little they care about copyright) from eachother, without sharing any competition need to get their own and re-crawl to update it too
Re: Gentoo bugzilla closed due AI bot scraper overload
#96While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome. Our largest offenders seems to be mostly…
You need to account for the fact that a lot of scraping is delegated by the big players to smaller players who can take loss of reputation and can use questionable methods (residential IPs). Some of the scrapers from these AI companies have been written very poorly from performance perspective. From what I understand the pressure has created even Google to be a lot more aggressive than what it was before. Not entirel…
So it could just be 1 guy with scrapy and infinite vc money
Re: Gentoo bugzilla closed due AI bot scraper overload
#97[dead]
Re: Gentoo bugzilla closed due AI bot scraper overload
#98(Not saying that it is the only solution / the right solution)
Would fronting it with Anubis have helped at all here?
(please do not ramble about how requiring javascript is a violation of your human rights)
Re: Gentoo bugzilla closed due AI bot scraper overload
#99There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable. We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time. Most scrapers are relatively honest in some way shape or form.
How did you implement this? My site's getting hammered, any tips would be appreciated
[flagged]
Re: Gentoo bugzilla closed due AI bot scraper overload
#100Earlier quoted context omitted.
That's the thing, they aren't.
Gentoo still has an active IRC and mailing-list. Back to basics? Then scrapers just see the static archives from, MARC I guess.
Gentoo has pretty much always had a Bugzilla, even back in the early 00s... there never was an alternative plan.
The biggest thing about what's happened today though, is how a "janitor" on the infrastructure team can shut down the entire distros ability to work on bugs. The decision to shut down should have been left to someone else like robbat.
Mgorny has consistently caused issues for Gentoo, and many people have left the community after having to deal with him, but it seems comrel/devrel can't/won't remove him for whatever reason.
Edit: it looks like bgo is up now. Hopefully it stays that way.