I had a lot of goodwill for GitHub but I think today is the tipping point. Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again. The hope is dead.
I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?
Incident with Github.com [resolved]
781–790 of 1001 posts
Re: Incident with Github.com [resolved]
#782I had a lot of goodwill for GitHub but I think today is the tipping point. Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again. The hope is dead.
I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?
Re: Incident with Github.com [resolved]
#783Earlier quoted context omitted.
It is interesting, how much money is being lost during this outage? My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced…
Is responding calmly a privilege to have, or a kindness to yourself?
Re: Incident with Github.com [resolved]
#784Earlier quoted context omitted.
> Luckily we already have a distributed version control system that can take over github Surprised that this wasn't just a link to Git. The biggest benefit of Git over predecessors was that it enabled decentralization. Github re-centralized the common case for the sake of convenience.
The biggest benefits of Git over predecessors were branching, 3-way merge, and history rewrites (in particular, rebase and squash). As much as Git fans wanted decentralization to be a major selling point, decentralization has never been a major selling point.
Re: Incident with Github.com [resolved]
#785> Update 16:59 UTC - The degradation affecting API Requests, Actions, Git Operations, Issues, Pages, Pull Requests and Webhooks has been mitigated. We are monitoring to ensure stability.
> Update 17:30 UTC - Git Operations is experiencing degraded performance. We are continuing to investigate.
> Update 17:36 UTC - Issues is experiencing degraded performance. We are continuing to investigate.
> Update 18:48 UTC - API Requests is experiencing degraded availability. We are continuing to investigate.
Re: Incident with Github.com [resolved]
#786Re: Incident with Github.com [resolved]
#787Re: Incident with Github.com [resolved]
#788I think for Github to redeem itself in the community eyes, it should come out with a very detailed technical analysis (short of proprietary details) what went wrong technically and what went wrong from a management perspective. I think it is hard to believe the line about the LLM influx at this point. I think nothing short of a full postmortem break down will restore trust in Github at this point. Even then it Github…
Re: Incident with Github.com [resolved]
#789Does anyone get the feeling that Microsoft is trying to kill GitHub in favor of ADO?
Re: Incident with Github.com [resolved]
#790We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?" No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it. What's everyone else using? Self hosted Gitlab? Gitea?
ultimately, folks need to stop writing massive YAML files to define their builds. CI should schedule jobs, your build system should define them. my workflow files look like this: steps: name: Do the thing run: nix build .# at any point, if my CI is down or i need to switch or i need to run builds locally, i can do so trivially.