Live data from Hacker News

Updated rate limits for unauthenticated requests

github.blog

171–180 of 187 posts

Re: Updated rate limits for unauthenticated requests

#171

Are the scraper sites using a large number of IP addresses, like a distributed denial of service attack? If not, rather than explicit blocking, consider using fair queuing. Do all the requests from IP addresses that have zero requests pending. Then those from IP addresses with one request pending, and so forth. Each IP address contends with itself, so making massive numbers of requests from one address won't cause a…

Huh, that sounds very reasonable, and it's the first time I've heard it mentioned. Why isn't this more wide-spread?

Because it doesn't help against DDOS attacks, with bogus request sources.

It's a good mitigation when you have legit requests, and some requestors create far more load than others. If Github used fair queuing for authenticated requests, heavy users would see slower response, but single requests would be serviced quickly. That tends to discourage overdoing it.

Still, if "git clone" stops working, we're going to need a Github alternative.

Re: Updated rate limits for unauthenticated requests

#172
post #167

Earlier quoted context omitted.

The moral argument in favor of piracy was that it didn’t cost the companies anything and the uses were noncommercial. Neither of those applies to the AI scrapers - they’re aggressively overusing freely-provided services (listen to some of the other folks on this thread about how the scrapers behave) and they’re doing so to create a competing commercial products. I’m not arguing you shouldn’t be annoyed by these chang…

There are plenty of commercial pirates, and those commercial uses were grouped in with noncommercial sharing in much the same way you are doing with scraping. Am I wrong in assuming most of this scraping comes from people utilizing AI agents for things like AI-assisted coding? If an AI agent scrapes a page at a users' request (say the 1 billionth git commit scraped today), do you consider that "loot[ing] a store"? Wh…

> There are plenty of commercial pirates, and those commercial uses were grouped in with noncommercial sharing

I don't think many people were particularly sympathetic to people making money off piracy - by and large, people were upset because people committing piracy for personal use were getting hit with the kinds of fines and legal charges usually reserved for, well, people who make money off piracy.

> Am I wrong in assuming most of this scraping comes from people utilizing AI agents for things like AI-assisted coding?

Yes. The huge increases in traffic aren't from, say, Claude going and querying Github when you ask it to, it's from the scraping to drive the initial training process. Claude and the others know the first thing about code because Github and StackOverflow were part of their training corpus, because the companies which made them scraped the whole damn site and used it as part of their training data for making a ~competing product. That's what Github's reacting to, that's what Reddit reacted to, that's what everyone's been reacting to - it's the scraping of the data for training that's leading to these reactions.

To be clear, because I think this is maybe a core of our disagreement: The problem that's leading to this isn't LLM agents acting on behalf of a user - it's not that Cursor googled python code for you - it's that the various companies training the models are aggressively scraping everything they can get their hands on. It's not one request for one repo on behalf of one user, it's the wholesale scraping of everything on the site by a rival company to make a rival product, most likely in violation of terms of service and certainly in violation of anything that anyone could reasonably assume another corporate entity would stand for. Github's not mad at you, they're mad at OpenAI.

> There is tons of software on Github written by users who wish for their work to remain open access. Is that the class of software and people you believe should be shuffled around into smaller and smaller services that haven't yet abandoned the commitments that allowed them to become popular?

You store your money in a bank. The bank gets robbed repeatedly by an organized group of serial bank robbers, and increases security at the branch. You move your money to another bank, because the increased security annoys you. You understand the problem here may repeat itself elsewhere as well, right?

Re: Updated rate limits for unauthenticated requests

#173

Earlier quoted context omitted.

True. I am simply wondering -- is the resistance from executives' so powerful that it can be never overpowered? Can't we ever just tell them "Look, this is like your car with plastic suspension -- it will work for a few days or even months but we can't rely on it forever; it's time to do it proper"? Especially when the car's plastic suspension is costing them extra money? I don't get it here, for real. I would think…

I'm fully with you. However, execs I know lease cars, not buy them, for that exact reason. You don't care if the suspension is made of plastic, if it's a subscription model. The metaphor very much falls apart but I had a point somewhere.

Oh yeah. That. :/

Well, there's a solution for that as well: execs should be liable for a number of years even after they move on. Para-troopers that swoop in, reap rewards they never worked for, and parachute away with the gold is something that must be legislated against, hard and aggressive. People should go to jail.

But... these are the people who make the rules so not happening, right?

Oh well, better luck to us in the next life I guess.

Re: Updated rate limits for unauthenticated requests

#174
post #128

Earlier quoted context omitted.

Yes, it's not the rate limits that are the problem per se but GitHub's tendency to log you out and make you go through 2fa. If they would let me stay logged in for a year then I wouldn't care so much.

You might be afflicted with some SSO or enterprise thing, I haven't logged into Github on my personal account in years.

Nope, just normal GitHub account.

Though GitHub did force me to use 2fa earlier because they said I have a "popular repo", so perhaps my account is considered high risk. Or maybe it's triggered by travelling and changing IP locations? I have no clue, but it's annoying to have to 2fa more than once in a blue moon.

Re: Updated rate limits for unauthenticated requests

#175

Earlier quoted context omitted.

Yes I actually browse GitHub anonymously. Not always but I do it every day. Never once had a problem. In another browser I log in because I do work with code in GitHub frequently. I comment on issues and PRs and all the normal stuff. I regularly drive two browsers, yes. I alternate between them multiple times per minute, often. In one, I am not logged in. In the other, I am logged in. Not once have I hit any anonymou…

> Not once have I hit any anonymous rate limit. I have a really hard time believing you on this. There's visual evidence from a year ago and it's consistent with my experience. And no, I haven't been hammering their servers. https://imgur.com/a/github-search-gated-behind-login-BT6uRIe > “I could do more in the past!” So what? So, I'll repeat what I said in the first comment that you replied to. GitHub captured the op…

[dead]

Re: Updated rate limits for unauthenticated requests

#176

Earlier quoted context omitted.

Are all contributors to open source under a lifetime obligation to never change their level of investment? Kind of a rhetorical question I guess, for a while I maintained a small open source project and yes, I still get entitled “why did you even publish this if you’re not going to fix the bug I reported” comments. Like, sorry, but my life priorities changed over the intervening 15 years. Fork it and fix it.

Microsoft didn't just give, they're benefitting massively from open source. And they're looking to extract even more value through data mining from forced logins and stealing GPL licensed code by laundering it using AI. There's no room for sympathy here.

There’s no sympathy in business. It’s a straw man to claim I’m looking for some emotional response.

But there is obligation. I’m asking if contributing to open source creates an obligation to do so forever, either for individuals or companies.

Re: Updated rate limits for unauthenticated requests

#177

Earlier quoted context omitted.

it was never about avoid scrapers. that's just the excuse. they own the scrapers too, remember. why do you think before they blocked non logged in users from even searching? they need your data and they are getting it exactly in their terms. because as I've said, they have already won.

[flagged]

> What the hell are … no, this is not a drug. This is a mental illness. Get help.

This is an unacceptable comment on HN and we have to ban accounts that do it repeatedly. We've warned you in the past about inappropriate comments. Please remind yourself of the guidelines and take care to observe them in future.

https://news.ycombinator.com/newsguidelines.html

Re: Updated rate limits for unauthenticated requests

#178

Earlier quoted context omitted.

Complex, stateful. I'm not even sure what that would look like for a huge service like GitHub. Where do you hold those many thousands of concurrent http connections and their pending request queues in a way that you can make decisions on them while making more operational sense than a simple rate limit? A lot of things would be easy if it were viable to have one big all-knowing giga load balancer. I remember Rap Geni…

A giga load balancer is no less viable than a giga Redis cache or a giga database. Rate limiting is inherently stateful - you can't rate limit a request without knowledge of prior requests, and that knowledge has to be stored somewhere . You can shift the state around, but you can't eliminate it. Sure, some solutions tend to be more efficient than others, but those typically boil down to implementation details rather…

[deleted]

Re: Updated rate limits for unauthenticated requests

#179

Earlier quoted context omitted.

I'm fully with you. However, execs I know lease cars, not buy them, for that exact reason. You don't care if the suspension is made of plastic, if it's a subscription model. The metaphor very much falls apart but I had a point somewhere.

Oh yeah. That. :/ Well, there's a solution for that as well: execs should be liable for a number of years even after they move on. Para-troopers that swoop in, reap rewards they never worked for, and parachute away with the gold is something that must be legislated against, hard and aggressive. People should go to jail. But... these are the people who make the rules so not happening, right? Oh well, better luck to us…

I found the best way to improve the product, as a programmer, is to know when you can squeeze in little refactors and improvements. My bosses learn to appreciate this pretty quickly.

Re: Updated rate limits for unauthenticated requests

#180

Earlier quoted context omitted.

Embrace, extend, extinguish.

… I… what has been embraced, extended and extinguished? I see no MS or GitHub specific extension, here. Copilot exists, and so do many other tools. Copilot can use lots of non-Microsoft models, too, including models from non-Microsoft companies. You can also get git repository hosting from other companies. You can even do it yourself. So, explain yourself. What has been embraced, extended, and extinguished? Be specif…

the entire open source community exist in github.

Microsoft have a more successful social network for programmers than HN or google circles (heh) ever dreamed.

the arguments had already dropped access to the information by scrapers, since they own the scrapers and all... why did you brought it back as the main argument? they hijacked what could have been a community hub and turned into a walled garden to sell a few enterprise licenses.

Post reply on HN