Most of these unauthenticated requests are read-only.
All of public github is only 21TB. Can't they just host that on a dumb cache and let the bots crawl to their heart's content?
Updated rate limits for unauthenticated requests
91–100 of 187 posts
Re: Updated rate limits for unauthenticated requests
#92Most of these unauthenticated requests are read-only. All of public github is only 21TB. Can't they just host that on a dumb cache and let the bots crawl to their heart's content?
I guess you're getting the size from the Arctic Code Vault? https://github.blog/news-insights/company-news/github-archiv... That was 5 years ago and is presumably in git's compressed storage format. Caching the corresponding GitHub HTML would take significantly more.
Re: Updated rate limits for unauthenticated requests
#93Earlier quoted context omitted.
I don't know if I ever recall seeing a CEO go to jail for practically anything, ever. I'm sure there are lots of examples, but at this point in my life I have kind of derived a rule of thumb of "if you want to commit a crime, just disguise it as a legitimate business" based off seeing so many times where CEOs get off scott free .
FWIW, Joseph Nacchio (the CEO of Qwest Communications) went to jail for like a decade for refusing to help the NSA violate FISA circa 2001.
He went to jail because he refused to help the NSA violate FISA, and then he sold the stock knowing that his refusal would cause the stock to drop (i.e. insider trading). His conviction was entirely on the basis of insider trading. The SEC went after him, not the NSA or DOJ or whatever.
Re: Updated rate limits for unauthenticated requests
#94Earlier quoted context omitted.
FWIW, Joseph Nacchio (the CEO of Qwest Communications) went to jail for like a decade for refusing to help the NSA violate FISA circa 2001.
He went to jail because he refused to help the NSA violate FISA, and then he sold the stock knowing that his refusal would cause the stock to drop (i.e. insider trading). His conviction was entirely on the basis of insider trading. The SEC went after him, not the NSA or DOJ or whatever.
> His conviction was entirely on the basis of insider trading. The SEC went after him, not the NSA or DOJ or whatever.
The SEC has no criminal prosecution powers; all they can do in that regard is write a note asking the DOJ to pretty-please look into something. The only way to get a federal (civilian) criminal conviction is to have the DOJ go after you.
Re: Updated rate limits for unauthenticated requests
#95Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…
Only they haven't started doing this right now. For many years, GitHub has been crippling unauthenticated browsing, doing it gradually to gauge the response. When unauthenticated, code search doesn't work at all and issue search stops working after like, 5 clicks at best. This is egregious behavior because Microsoft hasn't been upfront about this while they were doing this. Many open source projects are probably unaw…
Just sign in, problem solved. It baffles me that a site can provide a useful service that costs money to run, and all you need to do to use it is create a free account -- and people still find that egregious.
Re: Updated rate limits for unauthenticated requests
#96Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…
> Several people in the comments seem to be blaming Github for taking this step for no apparent reason.
I mean...
* Github is owned by Microsoft.
* The reason for this are AI crawlers.
* The reason AI crawlers exist in masses is an absurd hype around LLM+AI technology.
* The reason for that is... ChatGPT?
* The main investor of ChatGPT happens to be...?
Re: Updated rate limits for unauthenticated requests
#97Earlier quoted context omitted.
Only they haven't started doing this right now. For many years, GitHub has been crippling unauthenticated browsing, doing it gradually to gauge the response. When unauthenticated, code search doesn't work at all and issue search stops working after like, 5 clicks at best. This is egregious behavior because Microsoft hasn't been upfront about this while they were doing this. Many open source projects are probably unaw…
Just sign in, problem solved. It baffles me that a site can provide a useful service that costs money to run, and all you need to do to use it is create a free account -- and people still find that egregious.
That's not how consent works. GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible.
Re: Updated rate limits for unauthenticated requests
#98Earlier quoted context omitted.
Gitea’s been great, but I think a lot of its development has moved to Forgejo: https://forgejo.org/ That’s what I run on my personal server now.
I'm stuck on the latest gitea (1.22) that still supports migration to forgejo and unsure where to go next. So I've been following both projects (somewhat lazily), and it seems to me that gitea has the edge on feature development. Forgejo promised — but is yet to deliver any — interesting features like federation; meanwhile the real features they've been shipping are cosmetic changes like being able to set pronouns in…
I've successfully migrated from Gitea 1.23 by just rolling back migrations manually in SQL to where forgejo supports it again. Of course, I had backups.
Re: Updated rate limits for unauthenticated requests
#99Earlier quoted context omitted.
> Or maybe your IP/browser is questionable. I'm using Firefox and Brave on Linux from a residential internet provider in Europe and the 429 error triggers consistantly on both browsers. Not sure I would consider my setup questionable considering their target audience.
I’m browsing from an iPhone in Europe right now and can browse source code just fine without being logged in.
Then it means they're looking at the User-Agent string and determining that an iPhone in Europe most likely has a human using it, and might not require rate-limiting.
Re: Updated rate limits for unauthenticated requests
#100Earlier quoted context omitted.
What's being strip mined is the openness of the Internet, and AI isn't the one closing up shop. Github was created to collaborate on and share source code. The company in the best position to maximize access to free and open software is now just a dragon guarding other people's coins. The future is a .txt file of John Carmack pointing out how efficient software used to be, locked behind a repeating WAF captcha, forev…
AI isn't the one closing up shop, it’s the one looting all the stores and taking everything that isn’t bolted down. The AI companies are bad actors that are exploiting the openness of the internet in a fashion that was obviously going to lead to this result - the purpose of these scrapers is to grab everything they can and repackage it into a commercial product which doesn’t return anything to the original source. Of…
This rings the same tune as the MPAA and RIAA utilizing lawfare to destroy freedom online when pirates were the ones "break[ing] the internet."
Could you help me understand what the difference is between your point and the arguments MPAA and RIAA used to ruin the torrent users' lives they concluded were "thieves"?
As a rule of thumb, do you think people who are happy with the services they contribute content to being open access and wish them to remain so should be the ones who are forced to constantly migrate to new services to keep their content free?
When AI can perfectly replicate the browsing behavior of a human being, should Github restrict viewing a git repository to those who have verified blood biometrics or had their eyes scanned by an Orb? If they make that change, will you still place blame on "jackasses"?