Updated rate limits for unauthenticated requests • Hacker News Board

Re: Updated rate limits for unauthenticated requests

#131

Post by haiku2077 » Thu, May 15, 2025, 12:26 PM UTC

sph wrote: post #119

internetter wrote: post #79

Earlier quoted context omitted.

Big companies have thrown robots.txt to the wind when it comes to their precious AI models.

Yeah, they have openly disregarded copyright law, it's not a puny robots.txt file that's gonna stop them.

robots.txt isn't just an on/off switch. You can set crawler rate limits in there that crawlers may choose to respect, and the big companies respect them- because it's in their interest to reduce their crawling cost and not send more requests than they need to.

However, these smaller companies are doing ridiculous things like scraping the same site many thousands of times a day, far more often than the content of the sites change.

Re: Updated rate limits for unauthenticated requests

#132

Post by jrochkind1 » Thu, May 15, 2025, 1:00 PM UTC

This means it's no longer safe to point to github-hosted repos in `git:` or `github:` dependencies in ruby bundler, yes?

I forget because I don't use them, but weren't there some products meant as dependency package repositories that github had introduced at some point, for some platforms? Does this apply to them? (I would hope not unless they want to kill them?)

This rather enormously changes github's potential place in ecosystems.

What with the poor announcement/rollout -- also unusual for what we expect of github, if they had realized how much this effects -- I wonder if this was an "emergency" thing not fully thought out in response to the crazy decentralized bot deluge we've all been dealing with. I wonder if they will reconsider and come up with another solution -- this one and the way it was rolled out do not really match the ingenuity and competence we usually count on from github.

I think it will hurt github's reputation more than they realize if they don't provide a lot more context, with suggested workarounds for various use cases, and/or a rollback. This is actually quite an impactful change, in a way that the subtle rollout seems to suggest they didn't realize?

Re: Updated rate limits for unauthenticated requests

#133

Post by gs17 » Thu, May 15, 2025, 1:01 PM UTC

TheNewsIsHere wrote: post #23

I don’t think the publication date (May 8, as I type this) on the GitHub blog article is the same date this change became effective. From a long-term, clean network I have been consistently seeing these “whoa there!” secondary rate limit errors for over a month when browsing more than 2-3 files in a repo. My experience has been that once they’ve throttled your IP under this policy, you cannot even reach a login page…

Maybe they expect you to file the ticket from a different IP.

Re: Updated rate limits for unauthenticated requests

#134

Post by naikrovek » Thu, May 15, 2025, 1:14 PM UTC

mijoharas wrote: post #39

gnabgib wrote: post #3

60 req/hour for unauthenticated users 5000 req/hour for authenticated - personal 15000 req/hour for authenticated - enterprise org According to https://docs.github.com/en/rest/using-the-rest-api/rate-limi... I bump into this just browsing a repo's code (unauth).. seems like it's one of the side effects of the AI rush.

Why would the changelog update not include this? it's the most salient piece of information. I thought I was just misreading it and failing to see where they stated what the new rate limits were, since that's what anyone would care about when reading it.

> Why would the changelog update not include this?

I don't know. The limits in the comment that you're replying to are unchanged from where they were a year ago.

So far I don't see anything that has changed, and without an explanation from GitHub I don't think we'll know for sure what has changed.

Re: Updated rate limits for unauthenticated requests

#135

Post by naikrovek » Thu, May 15, 2025, 1:16 PM UTC

1oooqooq wrote: post #80

mijoharas wrote: post #39

Earlier quoted context omitted.

Why would the changelog update not include this? it's the most salient piece of information. I thought I was just misreading it and failing to see where they stated what the new rate limits were, since that's what anyone would care about when reading it.

because it will go way lower soon. and because they don't have to. they already have all your code. they've won.

you are not ... you don't have any part of your body in reality, do you? you have left the room.

If people training LLMs are excessively scraping GitHub, it is well within GitHub's purview to limit that activity. It's their site and it's up to them to make sure that it stays available. If that means that they curtail the activity of abusive users, then of course they're going to do that.

Re: Updated rate limits for unauthenticated requests

#136

Post by naikrovek » Thu, May 15, 2025, 1:19 PM UTC

soraminazuki wrote: post #97

jonas21 wrote: post #95

Earlier quoted context omitted.

Just sign in, problem solved. It baffles me that a site can provide a useful service that costs money to run, and all you need to do to use it is create a free account -- and people still find that egregious.

That's not how consent works. GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible.

> GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible.

It still is "open to all", but you can't abuse the service and expect to retain the ability to abuse the service.

Also where is "silently" coming from? This whole HN page is because someone linked to an article announcing the change...

I'm not really a fan of Microsoft anymore, but some of you have (apparently long ago) turned the corner into "anything Microsoft does that I don't want Microsoft to do is clearly Microsoft being evil" and that is simply not a reality-based viewpoint. sometimes Microsoft is doing something which one could consider "evil", but without knowledge that something evil is happening, you're assuming that evil is happening, and that's not really a valid way to think about things if you want to be heard by anyone.

Re: Updated rate limits for unauthenticated requests

#137

Post by DonHopkins » Thu, May 15, 2025, 1:36 PM UTC

dmitrygr wrote: post #123

miohtama wrote: post #104

Earlier quoted context omitted.

Criminally charging Russian and Chinese does not work. The solution would be to drop these contries off the internet if we want to play hard. The US cannot even stop NSO to hack the system with spyware and Israel is a political ally.

Most of ML training crawlers hitting my site come from USA. Should we drop USA too?

"US Out of North America!" -Ironic Protest T-Shirt and Graffiti (Social-Revolutionary Anarchist Federation) ;)

https://the-t-shirt-chronicles.com/2022/02/15/u-s-out-of/

Re: Updated rate limits for unauthenticated requests

#138

Post by brookst » Thu, May 15, 2025, 1:51 PM UTC

graemep wrote: post #125

whitehexagon wrote: post #109

If a company the size of MS isn't able handle the DOS caused by the LLM slurpers, then it really is game over for the open internet. We are going to need government approved ID based logins to even read the adverts at this rate. But this feels like a further attempt to create a walled garden around 'our' source code. I say our, but the first push to KYC, asking for phone numbers, was enough for me to delete all and c…

These exist, and you can self host. However, a lot of people think Github is the only option, and it benefits from network effects. Non-profit alternatives suffer from a lack of marketing and deal making. True of most things these days.

They also don’t have the resources to ensure perf and reliability if they get really popular, or to invest in UI and other goodness.

Still great for some applications and developers, but not all.

Re: Updated rate limits for unauthenticated requests

#139

Post by brookst » Thu, May 15, 2025, 1:57 PM UTC

soraminazuki wrote: post #97

jonas21 wrote: post #95

Earlier quoted context omitted.

Just sign in, problem solved. It baffles me that a site can provide a useful service that costs money to run, and all you need to do to use it is create a free account -- and people still find that egregious.

That's not how consent works. GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible.

Are all contributors to open source under a lifetime obligation to never change their level of investment?

Kind of a rhetorical question I guess, for a while I maintained a small open source project and yes, I still get entitled “why did you even publish this if you’re not going to fix the bug I reported” comments. Like, sorry, but my life priorities changed over the intervening 15 years. Fork it and fix it.

Re: Updated rate limits for unauthenticated requests

#140

Post by 1oooqooq » Thu, May 15, 2025, 2:23 PM UTC

naikrovek wrote: post #135

1oooqooq wrote: post #80

Earlier quoted context omitted.

because it will go way lower soon. and because they don't have to. they already have all your code. they've won.

you are not ... you don't have any part of your body in reality, do you? you have left the room. If people training LLMs are excessively scraping GitHub, it is well within GitHub's purview to limit that activity. It's their site and it's up to them to make sure that it stays available. If that means that they curtail the activity of abusive users, then of course they're going to do that.

it was never about avoid scrapers. that's just the excuse. they own the scrapers too, remember.

why do you think before they blocked non logged in users from even searching? they need your data and they are getting it exactly in their terms. because as I've said, they have already won.