Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Issues and Webhooks – Resolved

githubstatus.com

Post reply on HN
Print view View original

191–200 of 275 posts

Re: Incident with Issues and Webhooks – Resolved

#191

Post by galangalalgol »

stabbles wrote: post #154
wavemode wrote: post #104

Earlier quoted context omitted.

Git itself is kind of a fundamentally computationally inefficient way to store and retrieve information. If the problem to solve were simply "store and version this text", 14 billion commits in a year would not even be considered a lot. In other words, a centralized version control system built from the ground up to operate at scale would do far more for scalability than anything GitHub could possibly do to optimize…

What are you referring to when you say it's "fundamentally computationally inefficient"? It's pretty efficient because it's content-addressed, plus optimizations to reduce storage and data transfer with packfiles.

I suspect they were referring to some of the things git allows for non centralized version control. There are simplifications if you just wanted a centralized system like cvs had.

Re: Incident with Issues and Webhooks – Resolved

#193

Post by Philpax »

tardedmeme wrote: post #117
baq wrote: post #69

Earlier quoted context omitted.

I literally laughed out then shed a tear, because I'd actually take three 8s today.

Three eights is more than a month of downtime every year. Today is the three eights.

This felt wrong to my intuition, but, no, you're right: (1-0.888) * 12 = 1.344.

Re: Incident with Issues and Webhooks – Resolved

#194

Post by kqp »

AlexB138 wrote: post #15

Github has published some incredible usage rate increase numbers, which they ascribe to the rise of agentic coding. At some point, they are going to have to change rate limits, cut free-tier usage, or find some other path to reducing load. It's clear that their infrastructure can't keep up with this significant increase, and it's unlikely that they're going to just absorb the increased costs themselves. Very curious…

A week ago GitHub published a blog post saying this, a day later GitHub execs were in HN comments repeating it, and just like that it’s common knowledge that GitHub’s steady reliability decline from the 2019 onward was actually caused not by the 2019 Microsoft integration, but by something that did not exist until 2023. PR works, y’all. Turns out the reason GitHub doesn’t work is because it’s just so good!

Re: Incident with Issues and Webhooks – Resolved

#195

Post by graypegg »

fastball wrote: post #182
graypegg wrote: post #36

Earlier quoted context omitted.

IMO, they're reaching the point of no return. I don't think they can horizontally-scale their way out of the hole they dug themselves unless they separate their free and paid infra maybe... which doesn't seem likely considering how their other infra changes are going. In the same way you need to be 10x better for someone to consider switching to your product, if you get 10x worse your competitors get a free 10x by ju…

How can they not? Surely at GitHub scale there isn't a single component where they were relying on vertical scaling?

For all of it's history (up to and including now possibly?) Github was a big Ruby on Rails monolith. [0] Obviously some things run in their own service, but I'm seeing the core github features fall apart which should be the features packed into the big monolith. If load is this much a problem, not being able to only vertically scale the processes that need the extra headroom is a big problem. Scaling horizontally by just throwing more machines at it, or at least cordoning-off some machines as "the ones that people actually pay for" is all I can think of for an application I can only describe as "accidentally working". Urgency is most-definitely high and that pushes decision making towards permanently-temporary patches instead of actual infra/architecture improvements.

[0] https://github.blog/engineering/architecture-optimization/bu...

Re: Incident with Issues and Webhooks – Resolved

#196

Post by shevy-java »

AlexB138 wrote: post #15

Github has published some incredible usage rate increase numbers, which they ascribe to the rise of agentic coding. At some point, they are going to have to change rate limits, cut free-tier usage, or find some other path to reducing load. It's clear that their infrastructure can't keep up with this significant increase, and it's unlikely that they're going to just absorb the increased costs themselves. Very curious…

It means that Skynet is winning.

What you described above will piss off and alienate even more people. Eventually there will be a critical threshold crossed. Microslop will be the first victim of Skynet 11.0 (I lost track of its current version but you can see how much damage is caused by AI in general now - this was the beginning of skynet. Except that it sucks).

Re: Incident with Issues and Webhooks – Resolved

#197

Post by spockz »

PhilipRoman wrote: post #153
wolfi1 wrote: post #95

Earlier quoted context omitted.

I wonder how many of those actions are really necessary

And how many of those actions do uncached downloads instead of building self-contained offline images... Speaking of which, I wonder if GitHub has implemented any HTTP interception for common mirror sites, like used by apt, etc.

Many downloads now go over https. Intercepting them would require having certificate for those domains. IIRC on the clouds the standard images do have a sources list that points to mirrors on the cloud’s network. I would only presume Github Actions runners have the same.

Not sure if something similar exists for NPM which is big for all things JS.

Re: Incident with Issues and Webhooks – Resolved

#198

Post by munk-a »

stabbles wrote: post #160
amluto wrote: post #52

Earlier quoted context omitted.

For literally decades, I’ve observed that there are systems that make each operation cheap and systems that work hard to scale out. The former frequently seems to wildly outperform the latter. GitHub, for example, seems to implement the main repository /pulls page as a search query, which is hinted at by the prefilled search bar and was mostly confirmed last week when the search backend failed and pull requests didn’…

I noticed the same https://news.ycombinator.com/item?id=47940213 . My working hypothesis is that, given that a filter was always required (prs and issues are likely rows in the same database with a bool property to distinguish them), someone thought it'd be good to use the search API uniformly. But search is on the derivative of the underlying data, in contrast to the specific APIs for listing issues and prs.

Working in an organization without a mono-repository I've actually found it extremely difficult to keep a tab on PRs and issues across multiple repositories. For a problem that should be resolved by a "For me" page that just lists out all your active incoming and outgoing PRs their multi-page solution involving search filters that often need to be reset feels extremely weak. I've worked on large multi-tenant solutions before and a page where you can "SELECT * FROM everything LIMIT 10" is the absolute last thing you want to give to users.

It is bizarre to me that so much of their tooling defaults to acting across the whole of github data points without guiding the user towards (or even making available as far as I can tell) a way to easily scope requests down outside of a complex search filter.

Re: Incident with Issues and Webhooks – Resolved

#199

Post by amluto »

mnky9800n wrote: post #73
amluto wrote: post #52

Earlier quoted context omitted.

For literally decades, I’ve observed that there are systems that make each operation cheap and systems that work hard to scale out. The former frequently seems to wildly outperform the latter. GitHub, for example, seems to implement the main repository /pulls page as a search query, which is hinted at by the prefilled search bar and was mostly confirmed last week when the search backend failed and pull requests didn’…

Are you telling me you don’t want a chat interface to greet you when you log in to GitHub?

That’s sort of orthogonal. But if GitHub actually invoked an LLM on initial page load, that would be about par for the course, and it would be amusing for GitHub to then complain that they’ve grown so quickly that their systems can’t keep up.

Re: Incident with Issues and Webhooks – Resolved

#200

Post by cedws »

AlexB138 wrote: post #15

Github has published some incredible usage rate increase numbers, which they ascribe to the rise of agentic coding. At some point, they are going to have to change rate limits, cut free-tier usage, or find some other path to reducing load. It's clear that their infrastructure can't keep up with this significant increase, and it's unlikely that they're going to just absorb the increased costs themselves. Very curious…

The disappointing thing is that if you do some digging, you'll find the majority of that it's slop and just outright spam. There's a page on GitHub where you can see recently updated repositories and it's very rare I see anything of quality on there.

GitHub has become a dumping ground for broken code and it has more bots than ever. As much as I hate ID verification it might be a necessarily evil at this point because clearly their anti-bot measures aren't working.

Post reply on HN