Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Outage Tracker: Is GitHub Cooked?

isgithubcooked.com

Post reply on HN
Print view View original

141–150 of 195 posts

Re: GitHub Outage Tracker: Is GitHub Cooked?

#141

Post by snovv_crash »

Uvix wrote: post #55
tempest_ wrote: post #40

Earlier quoted context omitted.

Our GitlabCE instance has been sitting in the racks for nearly two years with almost 100% uptime running on 10 year old xeons that have long since paid for themselves. Of course it is not free of all management but for our use case it is working. There are hiccups with the CI runners from time time but nothing major and we have another machine in another rack that serves as a backup which can be brought up ~ I know c…

We have the expertise. We don’t have the interest in fighting our internal “security” teams to keep running VMs.

Potato potato.

Expertise isn't just "Do we have a sysadmin?", it's also "Does the org enable them to get their job done?".

Re: GitHub Outage Tracker: Is GitHub Cooked?

#142

Post by flyingshelf »

tdg5 wrote: post #95

I’ve reached the point where I just feel a mourning kind of sadness for GitHub

I feel that way for open source and programming in general. I don’t think it's going to be a thing anymore in just a couple of years. Even now I see people already don't look at any of the code they release.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#143

Post by chanux »

djieidj283 wrote: post #38

It’s ironic that the SCM that 2026 software engineers landed on, is one with a complicated distributed usage model and is slow/down because of a centralised service

Apparently it's not exactly git side of GitHub that's failing. It's all the bells and whistles bolted on later causing the trouble.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#144

Post by fulafel »

jeremyjh wrote: post #100
Bratmon wrote: post #96

Earlier quoted context omitted.

I wonder if this is a Microsoft Excel situation where everyone 1. Hates the status quo 2. Only uses a handful of the 10,000 features the status quo has 3. Uses a different handful than everyone else Because that model kept Excel as the center of buisnesses around the world for 30 years and counting

People don't hate Excel. Well - maybe programmers who don't use it hate it. In reality, Excel is the only reason Microsoft still exists. Excel is how they lure you in. Excel really is the best spreadsheet, its the only one every doc you are sent works with, but also the one people know inside and out. And if you are into Pivot tables, PowerQuery, etc there is just no escaping it. You are utterly stuck. But here is ho…

Microsoft has many products that people hate but are still the reason why they exist (eg Windows, Teams, Sharepoint etc).

Re: GitHub Outage Tracker: Is GitHub Cooked?

#146

Post by jeltz »

firesteelrain wrote: post #133
hansvm wrote: post #101

Earlier quoted context omitted.

The phrasing could have been better, but we've talked as a team about cancelling a $100k/yr contract nearly half a dozen times the last year, and the only thing keeping us from doing so is various "compliance" issues, and to a lesser degree the friction from physically moving. That's a tenuous moat, and if I were a GH PM I'd probably want to know that the current instability is somewhere near critical mass.

You are certainly free to attempt to migrate your workloads to your own or leased data center space in your area then buy your own on prem cloud like Dell Private Cloud and deal with all of the maintenance yourself. I don’t think it would amount to $100k/yr and you would be taking on all the risk.

It would be way less and risk would be lower (It is hard to reach one 9 no matter how bad you are at it), and self hosting is not the only option.

Re: GitHub Outage Tracker: Is GitHub Cooked?

#147

Post by idkasam »

linkregister wrote: post #130
hansvm wrote: post #101

Earlier quoted context omitted.

The phrasing could have been better, but we've talked as a team about cancelling a $100k/yr contract nearly half a dozen times the last year, and the only thing keeping us from doing so is various "compliance" issues, and to a lesser degree the friction from physically moving. That's a tenuous moat, and if I were a GH PM I'd probably want to know that the current instability is somewhere near critical mass.

For CI there are a number of drop-in commercially-available options. You can make it a staged migration, first trivially migrating the build runners, followed by the more complicated integration test and deployment runners. The CI harness itself can follow, and finally moving to a code hosting and visualization service is last. The final step is challenging; likely the most difficult part is changing all the code ref…

[dead]

Re: GitHub Outage Tracker: Is GitHub Cooked?

#148

Post by idkasam »

isityettime wrote: post #49
nightpool wrote: post #24

Getting rid of Actions and Copilot and other secondary services almost halves Github's incident rate: https://i.imgur.com/XPcMIFr.png I'm a big fan of Github Actions and I think people are often a little too harsh on it, but it's clear that it's sad that it's come at such a high cost to the platform's stability

GitHub Actions is really, really badly designed. The security model is fundamentally broken, the YAML hell is as bad as any, the log streaming lags like hell, they charge self-hosted runners for using their coordination plane, jobs queuing is really slow, and their software for actually running jobs is cursed and designed in a way that is practically hostile to self-hosting. GitHub Actions is god-awful. Have you ever…

[dead]

Re: GitHub Outage Tracker: Is GitHub Cooked?

#149

Post by flohofwoe »

kashnote wrote: post #12

I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI. I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figur…

> they're not limiting access to the site or (intentionally) throttling newcomers

They can simply throttle access for everyone to a push/merge frequency which was normal for human-maintained projects.

They should look at how MMOs battle cheating/botting and apply similar countermeasures (it's actually quite absurd that MMOs which are actively attacked every minute have a better uptimes than Github, and on Github the problems are even self-inflicted, not caused by external attacks).

But of course throttling bots to human-level push/merge frequency wouldn't be compatible with the Github leadership's 'AI bullishness'.

Post reply on HN