Updated rate limits for unauthenticated requests • Hacker News Board

Re: Updated rate limits for unauthenticated requests

#141

Post by PaulDavisThe1st » Thu, May 15, 2025, 2:24 PM UTC

VladVladikoff wrote: post #78

PaulDavisThe1st wrote: post #43

Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…

Have you noticed significant slowdown and CPU usage from failban with that many banned IPs? I saw it becoming a huge resource hog with far less IPs than that.

Yeah, when we hit about 80-100k banned hosts, iptables causes issues.

There are versions of iptables available that apparently can scale to 1M+ addresses, but our approach is just to unban all at that point, and then let things accumulate again.

Since we because responding with 404 to all commit URLs, the rate of banned address accumulation has slowed down quite a bit.

Re: Updated rate limits for unauthenticated requests

#142

Post by PaulDavisThe1st » Thu, May 15, 2025, 2:25 PM UTC

knowitnone wrote: post #67

PaulDavisThe1st wrote: post #43

Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…

you mean AI crawlers from Microsoft, owners of Github?

I have no idea where they are from. I'd surprised if MS is using a network of 1M+ residential IP addresses, but they've surprised me before ...

Re: Updated rate limits for unauthenticated requests

#143

Post by 1oooqooq » Thu, May 15, 2025, 2:25 PM UTC

hannob wrote: post #96

PaulDavisThe1st wrote: post #43

Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…

> Several people in the comments seem to be blaming Github for taking this step for no apparent reason. I mean... * Github is owned by Microsoft. * The reason for this are AI crawlers. * The reason AI crawlers exist in masses is an absurd hype around LLM+AI technology. * The reason for that is... ChatGPT? * The main investor of ChatGPT happens to be...?

almost like we bomb children because a politician told us to think of the children. crazy.

Re: Updated rate limits for unauthenticated requests

#144

Post by 1oooqooq » Thu, May 15, 2025, 2:26 PM UTC

londons_explore wrote: post #62

PaulDavisThe1st wrote: post #43

Several people in the comments seem to be blaming Github for taking this step for no apparent reason. Those of us who self-host git repos know that this is not true. Over at ardour.org, we've passed the 1M-unique-IP's banned due to AI trawlers sucking our repository 1 commit at a time. It was killing our server before we put fail2ban to work. I'm not arguing that the specific steps Github have taken are the right one…

Surely most AI trawlers have special support for git and just clone the repo once?

not if you vibe coded your crawler

Re: Updated rate limits for unauthenticated requests

#145

Post by jnky » Thu, May 15, 2025, 2:49 PM UTC

brookst wrote: post #139

soraminazuki wrote: post #97

Earlier quoted context omitted.

That's not how consent works. GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible.

Are all contributors to open source under a lifetime obligation to never change their level of investment? Kind of a rhetorical question I guess, for a while I maintained a small open source project and yes, I still get entitled “why did you even publish this if you’re not going to fix the bug I reported” comments. Like, sorry, but my life priorities changed over the intervening 15 years. Fork it and fix it.

All contributors to open source are not created equal. It is different when a literal 3 trillion dollar company does it, thus demonstrating they were unworthy of the trust and goodwill put in them. They have the money, they have the cloud infrastructure, they are doing all kinds of scraping themselves.

Re: Updated rate limits for unauthenticated requests

#146

Post by 6031769 » Thu, May 15, 2025, 2:50 PM UTC

IsTom wrote: post #120

arkh wrote: post #105

Earlier quoted context omitted.

Who could have known that Microsoft would pull some shenanigans? Is 20 years too long ago to learn from then? Embrace. Extend. Extinguish. This has never gone away.

When github was getting popular it was not owned by MS.

When github did not pull this sort of shenanigans it was not owned by MS.

Re: Updated rate limits for unauthenticated requests

#147

Post by soraminazuki » Thu, May 15, 2025, 2:58 PM UTC

naikrovek wrote: post #136

soraminazuki wrote: post #97

Earlier quoted context omitted.

That's not how consent works. GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible.

> GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible. It still is "open to all", but you can't abuse the service and expect to retain the ability to abuse the service. Also where is "silently" coming from? This whole HN page is because someone linked to an article announcing the change... I'm not r…

I repeat, this didn't start today. It has been happening for years. And no, browsing a few files or searching for an issue or two, which they totally kick in the rate limit for, isn't "abuse."

Re: Updated rate limits for unauthenticated requests

#148

Post by sebmellen » Thu, May 15, 2025, 3:05 PM UTC

1oooqooq wrote: post #140

naikrovek wrote: post #135

Earlier quoted context omitted.

you are not ... you don't have any part of your body in reality, do you? you have left the room. If people training LLMs are excessively scraping GitHub, it is well within GitHub's purview to limit that activity. It's their site and it's up to them to make sure that it stays available. If that means that they curtail the activity of abusive users, then of course they're going to do that.

it was never about avoid scrapers. that's just the excuse. they own the scrapers too, remember. why do you think before they blocked non logged in users from even searching? they need your data and they are getting it exactly in their terms. because as I've said, they have already won.

Embrace, extend, extinguish.

Re: Updated rate limits for unauthenticated requests

#149

Post by soraminazuki » Thu, May 15, 2025, 3:06 PM UTC

brookst wrote: post #139

soraminazuki wrote: post #97

Earlier quoted context omitted.

That's not how consent works. GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible.

Are all contributors to open source under a lifetime obligation to never change their level of investment? Kind of a rhetorical question I guess, for a while I maintained a small open source project and yes, I still get entitled “why did you even publish this if you’re not going to fix the bug I reported” comments. Like, sorry, but my life priorities changed over the intervening 15 years. Fork it and fix it.

Microsoft didn't just give, they're benefitting massively from open source. And they're looking to extract even more value through data mining from forced logins and stealing GPL licensed code by laundering it using AI. There's no room for sympathy here.

Re: Updated rate limits for unauthenticated requests

#150

Post by hn276 » Thu, May 15, 2025, 3:26 PM UTC

naikrovek wrote: post #136

soraminazuki wrote: post #97

Earlier quoted context omitted.

That's not how consent works. GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible.

> GitHub captured the open source ecosystem under the premise that its code and issue tracker will remain open to all. Silently changing the deal afterwards is reprehensible. It still is "open to all", but you can't abuse the service and expect to retain the ability to abuse the service. Also where is "silently" coming from? This whole HN page is because someone linked to an article announcing the change... I'm not r…

[flagged]