Earlier quoted context omitted.
Yep. But AI trawlers don't use it. Ask them why.
Do we know it's AI trawlers? And not just generally degenerate bots? Or just one evil bot network?
Is there any difference between those 3?
Updated rate limits for unauthenticated requests
61–70 of 187 posts
Re: Updated rate limits for unauthenticated requests
#62Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…
Surely most AI trawlers have special support for git and just clone the repo once?
Re: Updated rate limits for unauthenticated requests
#63Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…
Surely most AI trawlers have special support for git and just clone the repo once?
The AI companies could do work or they could not do work.
They've pretty widely chosen to not do work and just slam websites from proxy IPs instead.
You would think their products would be used by them to do the work if they worked as well as advertised...
Re: Updated rate limits for unauthenticated requests
#64The truth is this won't actually stop AI crawlers and they'll just move to a large residential proxy pool to work around it. Not sure what the solution is honestly.
Criminal charges under CFAA to actual CEOs of actual companies doing this, with long jail terms.
Re: Updated rate limits for unauthenticated requests
#65Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…
Surely most AI trawlers have special support for git and just clone the repo once?
Apparently, the vibe coding session didn't account for it. /s
I would more readily assume a large social networking company filled with bright minds would have worked out some kind of agreement on, say, a large corpus of copyrighted training data before using it.
It's the wild wild west right now. Data is king for AI training.
Re: Updated rate limits for unauthenticated requests
#66you mean you want to better track users
Re: Updated rate limits for unauthenticated requests
#67Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…
you mean AI crawlers from Microsoft, owners of Github?
Re: Updated rate limits for unauthenticated requests
#68Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…
you mean AI crawlers from Microsoft, owners of Github?
The big companies tend to respect robots.txt. The problem is other, unscrupulous actors use fake user agents and residential IPs and don't respect robots.txt or act reasonably.
Re: Updated rate limits for unauthenticated requests
#69I assume they're trying to keep ai bots from strip mining the whole place. Or maybe your IP/browser is questionable.
What's being strip mined is the openness of the Internet, and AI isn't the one closing up shop. Github was created to collaborate on and share source code. The company in the best position to maximize access to free and open software is now just a dragon guarding other people's coins. The future is a .txt file of John Carmack pointing out how efficient software used to be, locked behind a repeating WAF captcha, forev…
AI isn't the one closing up shop, it’s the one looting all the stores and taking everything that isn’t bolted down. The AI companies are bad actors that are exploiting the openness of the internet in a fashion that was obviously going to lead to this result - the purpose of these scrapers is to grab everything they can and repackage it into a commercial product which doesn’t return anything to the original source. Of course this was going to break the internet, and people have been warning about that from the first moment these jackasses started - what the hell else was the outcome of all this going to be?
Re: Updated rate limits for unauthenticated requests
#70Earlier quoted context omitted.
Gitea’s been great, but I think a lot of its development has moved to Forgejo: https://forgejo.org/ That’s what I run on my personal server now.
I'm stuck on the latest gitea (1.22) that still supports migration to forgejo and unsure where to go next. So I've been following both projects (somewhat lazily), and it seems to me that gitea has the edge on feature development. Forgejo promised — but is yet to deliver any — interesting features like federation; meanwhile the real features they've been shipping are cosmetic changes like being able to set pronouns in…
That's an interesting perspective, and I can't strongly disprove it, but that doesn't match my impression. I cloned both repos (Gitea's from GitHub; Forgejo's from Codeberg, which runs on Forgejo) and ran this command:
git log --since="1 year ago" --format="%an" | sort | uniq -c | sort -n | wc -l
And it looks to me like Forgejo is using a similar process of combining lots of smaller PR commits into a single merge commit. The wide majority of its commits since last June or so seem to be 1-commit-per-PR. Changing the above command to `--since="2024-07-1"` reduces the number of unique contributors to 136 for Gitea, 217 for Forgejo. It also shows 1228 commits for Gitea and 3039 for Forgejo, and I do think that's a legitimately apples-to-apples comparison.
If we brute force it and run
git log --since="1 year ago" | rg '\(\#\d{4,5}\)' | wc -l
And finally, their respective activity pages (https://github.com/go-gitea/gitea/pulse/monthly and https://codeberg.org/forgejo/forgejo/activity/monthly) show a similar story.
I'm not an expert in methodology here, but from my initial poking around, it would seem to me that Forgejo has a lot more activity and variety of contributors than Gitea does.