Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Updated rate limits for unauthenticated requests

github.blog

Post reply on HN
Print view View original

151–160 of 187 posts

Re: Updated rate limits for unauthenticated requests

#151

Post by rendaw »

usernamed7 wrote: post #55
gnabgib wrote: post #3

60 req/hour for unauthenticated users 5000 req/hour for authenticated - personal 15000 req/hour for authenticated - enterprise org According to https://docs.github.com/en/rest/using-the-rest-api/rate-limi... I bump into this just browsing a repo's code (unauth).. seems like it's one of the side effects of the AI rush.

1 request a minute?!? wow that's just absurd you get it for just looking through code.

I opened a repo in a spare computer browser and clicked on a couple things and got a rate limit error. It feels effectively unusable unless you're logged in now (couldn't search from before, now you can't even browse).

Re: Updated rate limits for unauthenticated requests

#152

Post by grishka »

tonyhart7 wrote: post #101
grishka wrote: post #90

Earlier quoted context omitted.

What about AI not respecting robots.txt? I myself have never ran into this, but I've seen complaints of many people who did.

"What about AI not respecting robots.txt?" since when actor that want gather your entire data respect things like this??? how can you enforce such things with just "please don't crawl this directory thanks"

You can't enforce them, but for the entire preceding history of the internet, most crawlers respected them. But then AI happened and those companies decided that their noble mission of forcing their slop into as many facets of human life as possible is above some stupid rules.

Re: Updated rate limits for unauthenticated requests

#153

Post by naikrovek »

soraminazuki wrote: post #147
naikrovek wrote: post #136

Earlier quoted context omitted.

> GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible. It still is "open to all", but you can't abuse the service and expect to retain the ability to abuse the service. Also where is "silently" coming from? This whole HN page is because someone linked to an article announcing the change... I'm not r…

I repeat, this didn't start today. It has been happening for years. And no, browsing a few files or searching for an issue or two, which they totally kick in the rate limit for, isn't "abuse."

They don’t rate limit someone who is browsing with a normal usage pattern. They did for a day or two, then discovered their mistake and fixed it.

> years

No.

> a few

I’ve always considered “a few” to be “between 3 and 12” and 60 is more than “a few”.

Re: Updated rate limits for unauthenticated requests

#154

Post by naikrovek »

1oooqooq wrote: post #140
naikrovek wrote: post #135

Earlier quoted context omitted.

you are not ... you don't have any part of your body in reality, do you? you have left the room. If people training LLMs are excessively scraping GitHub, it is well within GitHub's purview to limit that activity. It's their site and it's up to them to make sure that it stays available. If that means that they curtail the activity of abusive users, then of course they're going to do that.

it was never about avoid scrapers. that's just the excuse. they own the scrapers too, remember. why do you think before they blocked non logged in users from even searching? they need your data and they are getting it exactly in their terms. because as I've said, they have already won.

[flagged]

Re: Updated rate limits for unauthenticated requests

#155

Post by naikrovek »

sebmellen wrote: post #148
1oooqooq wrote: post #140

Earlier quoted context omitted.

it was never about avoid scrapers. that's just the excuse. they own the scrapers too, remember. why do you think before they blocked non logged in users from even searching? they need your data and they are getting it exactly in their terms. because as I've said, they have already won.

Embrace, extend, extinguish.

… I… what has been embraced, extended and extinguished?

I see no MS or GitHub specific extension, here. Copilot exists, and so do many other tools. Copilot can use lots of non-Microsoft models, too, including models from non-Microsoft companies. You can also get git repository hosting from other companies. You can even do it yourself.

So, explain yourself. What has been embraced, extended, and extinguished? Be specific. No “vibes”. Cite your sources or admit you have none. I see no extending unique to MS and I see no extinguishing. So explain yourself.

Re: Updated rate limits for unauthenticated requests

#156

Post by roughly »

globie wrote: post #100
roughly wrote: post #69

Earlier quoted context omitted.

AI isn't the one closing up shop, it’s the one looting all the stores and taking everything that isn’t bolted down. The AI companies are bad actors that are exploiting the openness of the internet in a fashion that was obviously going to lead to this result - the purpose of these scrapers is to grab everything they can and repackage it into a commercial product which doesn’t return anything to the original source. Of…

This rings the same tune as the MPAA and RIAA utilizing lawfare to destroy freedom online when pirates were the ones "break[ing] the internet." Could you help me understand what the difference is between your point and the arguments MPAA and RIAA used to ruin the torrent users' lives they concluded were "thieves"? As a rule of thumb, do you think people who are happy with the services they contribute content to being…

The moral argument in favor of piracy was that it didn’t cost the companies anything and the uses were noncommercial. Neither of those applies to the AI scrapers - they’re aggressively overusing freely-provided services (listen to some of the other folks on this thread about how the scrapers behave) and they’re doing so to create a competing commercial products.

I’m not arguing you shouldn’t be annoyed by these changes, I’m arguing you should be mad at the right people. The scrapers violated the implicit contract of the open internet, and now that’s being made more explicit. GitHub’s not actually a charity, but they’ve been able to provide a free service in exchange for the good will and community that comes along with it driving enough business to cover their costs of providing that service. The scrapers have changed that math, as they did with every other site on the internet in a similar fashion. You can’t loot a store and expect them not to upgrade the locks - as the saying goes, the enemy gets a vote on your strategy, too.

Re: Updated rate limits for unauthenticated requests

#157

Post by soraminazuki »

naikrovek wrote: post #153
soraminazuki wrote: post #147

Earlier quoted context omitted.

I repeat, this didn't start today. It has been happening for years. And no, browsing a few files or searching for an issue or two, which they totally kick in the rate limit for, isn't "abuse."

They don’t rate limit someone who is browsing with a normal usage pattern. They did for a day or two, then discovered their mistake and fixed it. > years No. > a few I’ve always considered “a few” to be “between 3 and 12” and 60 is more than “a few”.

I'm speaking from direct experience over the past few years, from my home, work, and outside with a phone. Do you actually browse GitHub anonymously, or are you reflexively shifting blame?

If you need more proof, this is last year:

https://news.ycombinator.com/item?id=39322838

And this is another year before that:

https://news.ycombinator.com/item?id=36254129

Oh look, there's even visual proof in the discussion:

https://imgur.com/a/github-search-gated-behind-login-BT6uRIe

Re: Updated rate limits for unauthenticated requests

#158

Post by 10000truths »

hombre_fatal wrote: post #130
fellerts wrote: post #111

Earlier quoted context omitted.

Huh, that sounds very reasonable, and it's the first time I've heard it mentioned. Why isn't this more wide-spread?

Complex, stateful. I'm not even sure what that would look like for a huge service like GitHub. Where do you hold those many thousands of concurrent http connections and their pending request queues in a way that you can make decisions on them while making more operational sense than a simple rate limit? A lot of things would be easy if it were viable to have one big all-knowing giga load balancer. I remember Rap Geni…

A giga load balancer is no less viable than a giga Redis cache or a giga database. Rate limiting is inherently stateful - you can't rate limit a request without knowledge of prior requests, and that knowledge has to be stored somewhere. You can shift the state around, but you can't eliminate it.

Sure, some solutions tend to be more efficient than others, but those typically boil down to implementation details rather than fundamental limitations in system design.

Re: Updated rate limits for unauthenticated requests

#159

Post by naikrovek »

soraminazuki wrote: post #157
naikrovek wrote: post #153

Earlier quoted context omitted.

They don’t rate limit someone who is browsing with a normal usage pattern. They did for a day or two, then discovered their mistake and fixed it. > years No. > a few I’ve always considered “a few” to be “between 3 and 12” and 60 is more than “a few”.

I'm speaking from direct experience over the past few years, from my home, work, and outside with a phone. Do you actually browse GitHub anonymously, or are you reflexively shifting blame? If you need more proof, this is last year: https://news.ycombinator.com/item?id=39322838 And this is another year before that: https://news.ycombinator.com/item?id=36254129 Oh look, there's even visual proof in the discussion: http…

Yes I actually browse GitHub anonymously. Not always but I do it every day. Never once had a problem.

In another browser I log in because I do work with code in GitHub frequently. I comment on issues and PRs and all the normal stuff.

I regularly drive two browsers, yes. I alternate between them multiple times per minute, often. In one, I am not logged in. In the other, I am logged in.

Not once have I hit any anonymous rate limit.

I respect one’s desire to use something without logging in, that’s fine. But what you do when you use up the free tier of a service is one of the following: A) you pay for the next tier, B) you (in this case) log in so that your usage is no longer considered “anonymous”, or C) you wait for the next usage measurement period to begin so that you can resume.

It’s their service and they can decide how they want to provide it, in the exact same way that you can decide how to provide any services that you might provide.

If it is your privacy that you are considering by not having an account, fine. By making that choice you are limiting yourself to whatever the services you use decide to give you, and you are entitled to nothing.

“I could do more in the past!” So what? They decided to let you do more in the past, and now they’ve decided to let you do less. They don’t owe you free services; you choose to use the free service and by doing so you’ve chosen to be bound by any usage caps that they decide to apply to you.

Nobody owes you free services AT ALL, but you’re getting them anyway. Instead of feeling entitled to more than you’re getting, maybe be thankful for what you have.

Re: Updated rate limits for unauthenticated requests

#160

Post by soraminazuki »

naikrovek wrote: post #159
soraminazuki wrote: post #157

Earlier quoted context omitted.

I'm speaking from direct experience over the past few years, from my home, work, and outside with a phone. Do you actually browse GitHub anonymously, or are you reflexively shifting blame? If you need more proof, this is last year: https://news.ycombinator.com/item?id=39322838 And this is another year before that: https://news.ycombinator.com/item?id=36254129 Oh look, there's even visual proof in the discussion: http…

Yes I actually browse GitHub anonymously. Not always but I do it every day. Never once had a problem. In another browser I log in because I do work with code in GitHub frequently. I comment on issues and PRs and all the normal stuff. I regularly drive two browsers, yes. I alternate between them multiple times per minute, often. In one, I am not logged in. In the other, I am logged in. Not once have I hit any anonymou…

> Not once have I hit any anonymous rate limit.

I have a really hard time believing you on this. There's visual evidence from a year ago and it's consistent with my experience. And no, I haven't been hammering their servers.

https://imgur.com/a/github-search-gated-behind-login-BT6uRIe

> “I could do more in the past!” So what?

So, I'll repeat what I said in the first comment that you replied to. GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible.

> Instead of feeling entitled

Again, I'll just repeat yet another one of my comments. Microsoft didn't just give, they're benefitting massively from open source. And they're looking to extract even more value through data mining from forced logins and stealing GPL licensed code by laundering it using AI. Open source projects that chose GitHub didn't agree to this!

> be thankful for what you have

You can't be serious. Yeah, be grateful for the trillion dollar company buying a service it didn't create, extracting as much value as they can from it in questionable ways and tearing up social contract!

Post reply on HN