The blog post is tagged with "improvement" - ironic for more restrictive rate limits.
Also, neither the new nor the old rate limits are mentioned.
Updated rate limits for unauthenticated requests
71–80 of 187 posts
Re: Updated rate limits for unauthenticated requests
#7260 req/hour for unauthenticated users 5000 req/hour for authenticated - personal 15000 req/hour for authenticated - enterprise org According to https://docs.github.com/en/rest/using-the-rest-api/rate-limi... I bump into this just browsing a repo's code (unauth).. seems like it's one of the side effects of the AI rush.
1/min? That’s insanely low.
Re: Updated rate limits for unauthenticated requests
#73Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…
Surely most AI trawlers have special support for git and just clone the repo once?
I think you vastly overestimate the average dev and their care for handling special cases that are mostly other people’s aggregate problem.
Re: Updated rate limits for unauthenticated requests
#74Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…
Only they haven't started doing this right now. For many years, GitHub has been crippling unauthenticated browsing, doing it gradually to gauge the response. When unauthenticated, code search doesn't work at all and issue search stops working after like, 5 clicks at best.
This is egregious behavior because Microsoft hasn't been upfront about this while they were doing this. Many open source projects are probably unaware that their issue tracker has been walled off, creating headaches unbeknownst to them.
Re: Updated rate limits for unauthenticated requests
#75Even with authenticated requests, viewing a pull request and adding `.diff` to the end of the URL is currently ratelimited at 1 request per minute. Incredibly low, IMO.
This is going to make the job of package managers a PITA. Especially Nix.
Re: Updated rate limits for unauthenticated requests
#76The truth is this won't actually stop AI crawlers and they'll just move to a large residential proxy pool to work around it. Not sure what the solution is honestly.
Criminal charges under CFAA to actual CEOs of actual companies doing this, with long jail terms.
I don't know if I ever recall seeing a CEO go to jail for practically anything, ever. I'm sure there are lots of examples, but at this point in my life I have kind of derived a rule of thumb of "if you want to commit a crime, just disguise it as a legitimate business" based off seeing so many times where CEOs get off scott free .
Re: Updated rate limits for unauthenticated requests
#7760 req/hour for unauthenticated users 5000 req/hour for authenticated - personal 15000 req/hour for authenticated - enterprise org According to https://docs.github.com/en/rest/using-the-rest-api/rate-limi... I bump into this just browsing a repo's code (unauth).. seems like it's one of the side effects of the AI rush.
1/min? That’s insanely low.
60/hr is not the same as 1/min, unless you're trying to continually make as many requests as possible, like a crawler. and if that is for your use case, then your traffic is probably exactly what they're trying to block.
Re: Updated rate limits for unauthenticated requests
#78Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…
Have you noticed significant slowdown and CPU usage from failban with that many banned IPs? I saw it becoming a huge resource hog with far less IPs than that.
Re: Updated rate limits for unauthenticated requests
#79Earlier quoted context omitted.
you mean AI crawlers from Microsoft, owners of Github?
The big companies tend to respect robots.txt. The problem is other, unscrupulous actors use fake user agents and residential IPs and don't respect robots.txt or act reasonably.
Big companies have thrown robots.txt to the wind when it comes to their precious AI models.
Re: Updated rate limits for unauthenticated requests
#8060 req/hour for unauthenticated users 5000 req/hour for authenticated - personal 15000 req/hour for authenticated - enterprise org According to https://docs.github.com/en/rest/using-the-rest-api/rate-limi... I bump into this just browsing a repo's code (unauth).. seems like it's one of the side effects of the AI rush.
Why would the changelog update not include this? it's the most salient piece of information. I thought I was just misreading it and failing to see where they stated what the new rate limits were, since that's what anyone would care about when reading it.
because it will go way lower soon. and because they don't have to.
they already have all your code. they've won.