Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updated rate limits for unauthenticated requests

github.blog

Post reply on HN
Print view View original

111–120 of 187 posts

Re: Updated rate limits for unauthenticated requests

#111

Post by fellerts »

Animats wrote: post #108

Are the scraper sites using a large number of IP addresses, like a distributed denial of service attack? If not, rather than explicit blocking, consider using fair queuing. Do all the requests from IP addresses that have zero requests pending. Then those from IP addresses with one request pending, and so forth. Each IP address contends with itself, so making massive numbers of requests from one address won't cause a…

Huh, that sounds very reasonable, and it's the first time I've heard it mentioned. Why isn't this more wide-spread?

Re: Updated rate limits for unauthenticated requests

#112

Post by jclulow »

Animats wrote: post #108

Are the scraper sites using a large number of IP addresses, like a distributed denial of service attack? If not, rather than explicit blocking, consider using fair queuing. Do all the requests from IP addresses that have zero requests pending. Then those from IP addresses with one request pending, and so forth. Each IP address contends with itself, so making massive numbers of requests from one address won't cause a…

Yes, LLM-era scrapers are frequently making use of large numbers of IP addresses from all over the place. Some of them seem to be bot nets, but based on IP subnet ownership it seems also pretty frequently to be cloud companies, many of them outside the US. In addition to fanning out to different IPs, many of the scrapers appear to use User Agent strings that are randomised, or perhaps in some cases themselves generated by the slop factory. It's pretty fucking bleak out there, to be honest.

Re: Updated rate limits for unauthenticated requests

#113

Post by voidnap »

miohtama wrote: post #106
voidnap wrote: post #7

Earlier quoted context omitted.

I encountered this on github last week. Very agressive rate limiting. My browser and IP is very ordinary. Since Microsoft is struggling to make ends meet, maybe they could throw a captcha or proof of work like Anubis by xe iaso. They already disabled code search for unauthenticated users. Its totally plausible they will disable code browsing as well.

Just sign in if it's an issue for your usage.

My usage isn't high. I was rate limited to like 5 requests per minute. It was a repo with several small files.

And seriously if they keep this up, with limits on their web interface but leave unauthenticated cloning allowed, I'd rather clone the repo than log in.

GitHub code browsing went south since microsoft bought them anyway. Having a simple proxy that clones a repo and serves it would solve problems with rate limits and their awful UX.

Re: Updated rate limits for unauthenticated requests

#114

Post by ghssds »

whitehexagon wrote: post #109

If a company the size of MS isn't able handle the DOS caused by the LLM slurpers, then it really is game over for the open internet. We are going to need government approved ID based logins to even read the adverts at this rate. But this feels like a further attempt to create a walled garden around 'our' source code. I say our, but the first push to KYC, asking for phone numbers, was enough for me to delete all and c…

you mean https://savannah.gnu.org?

Re: Updated rate limits for unauthenticated requests

#115

Post by uallo »

PaulDavisThe1st wrote: post #43

Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…

That is also a problem on a side project I've been running for several years. It is based on a heavily rate-limited third-party API. And the main problem is that bots often cause (huge) traffic spikes which essentially DDoSes the application. Luckily, a large part of these bots can easily be detected based on their behaviour in my specific case. I started serving them trash data and have not been DDoSed since.

Re: Updated rate limits for unauthenticated requests

#116

Post by notpushkin »

ghssds wrote: post #114
whitehexagon wrote: post #109

If a company the size of MS isn't able handle the DOS caused by the LLM slurpers, then it really is game over for the open internet. We are going to need government approved ID based logins to even read the adverts at this rate. But this feels like a further attempt to create a walled garden around 'our' source code. I say our, but the first push to KYC, asking for phone numbers, was enough for me to delete all and c…

you mean https://savannah.gnu.org ?

Or maybe https://codeberg.org/.

Re: Updated rate limits for unauthenticated requests

#117

Post by Probiotic6081 »

mijoharas wrote: post #39
gnabgib wrote: post #3

60 req/hour for unauthenticated users 5000 req/hour for authenticated - personal 15000 req/hour for authenticated - enterprise org According to https://docs.github.com/en/rest/using-the-rest-api/rate-limi... I bump into this just browsing a repo's code (unauth).. seems like it's one of the side effects of the AI rush.

Why would the changelog update not include this? it's the most salient piece of information. I thought I was just misreading it and failing to see where they stated what the new rate limits were, since that's what anyone would care about when reading it.

[dead]

Re: Updated rate limits for unauthenticated requests

#118

Post by jashmatthews »

Animats wrote: post #108

Are the scraper sites using a large number of IP addresses, like a distributed denial of service attack? If not, rather than explicit blocking, consider using fair queuing. Do all the requests from IP addresses that have zero requests pending. Then those from IP addresses with one request pending, and so forth. Each IP address contends with itself, so making massive numbers of requests from one address won't cause a…

Exactly that. It's an arms race between companies that offer a large number of residential IPs as proxies and companies that run unauthenticated web services trying not to die from denial of service.

https://brightdata.com/

Re: Updated rate limits for unauthenticated requests

#119

Post by sph »

internetter wrote: post #79
haiku2077 wrote: post #68

Earlier quoted context omitted.

The big companies tend to respect robots.txt. The problem is other, unscrupulous actors use fake user agents and residential IPs and don't respect robots.txt or act reasonably.

Big companies have thrown robots.txt to the wind when it comes to their precious AI models.

Yeah, they have openly disregarded copyright law, it's not a puny robots.txt file that's gonna stop them.

Re: Updated rate limits for unauthenticated requests

#120

Post by IsTom »

arkh wrote: post #105
soraminazuki wrote: post #97

Earlier quoted context omitted.

That's not how consent works. GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible.

Who could have known that Microsoft would pull some shenanigans? Is 20 years too long ago to learn from then? Embrace. Extend. Extinguish. This has never gone away.

When github was getting popular it was not owned by MS.
Post reply on HN