In the old days we had mirrors for many online repositories.
GitHub having issues today
21–30 of 89 posts
Re: GitHub having issues today
#22Earlier quoted context omitted.
Full stack re-writes are not always the best way. Sometimes you end up with worse. Sometimes you end up with better. If you do go the 'full stack rewrite' you better have a decent plan in place. Because you are about to get to support 2 code bases for awhile. edit: fair enough
I'm pretty sure (hope?) the comment was said _in jest_ (though I'm not familiar with _this internet standard_)
Yeah. I thought markdown would work here.
Re: GitHub having issues today
#23Still having regular incidents at GitHub in 2024, even with Microsoft's infrastructure after 5 years since the acquisition with something always going down. Just expect GitHub to go down at least once every month as it is that unreliable. This certainly has aged well: [0] [0] https://news.ycombinator.com/item?id=22868406
The price teams will pay to offload their ops.
People really like avoiding ops
Re: GitHub having issues today
#24We have a slack channel that monitors GitHub availability. There’s content there nearly 3-4 times a week. It’s amazing how awful this has become.
Re: GitHub having issues today
#25Earlier quoted context omitted.
More likely, it's efforts to migrate away from the previously solid Rails stack to MS's preferred stack. They've had a long history of this kind of stability issue when migrating or trying to migrate acquisitions from their previous stack to an MS one. This happened with Hotmail (Unix server -> Windows server), LinkedIn (custom cloud -> MS cloud) and others since.
Is Github moving to .Net and/or SQLServer, or is it "just" moving everything to Azure?
The latter.
Re: GitHub having issues today
#26And just as we're about to migrate 4 kubernetes clusters with a total of ~4k pods. Terraform in github actions on selfhosted runners and argoCD is failing.
Oh that sucks, there's always going to be those who will say that it's the price you pay for using Github, but locally hosted VCS and CI/CD systems have issues as well. External dependencies are always problem, but do you have the capacity and resources required to manage those dependencies internally? Most don't and will still get a better product/service by using an external service.
Rate of outages on github last few years has been orders of magnitude higher than anything I've encountered on a locally hosted VCS.
Local also means you can orchestrate maintenance windows to avoid outages at critical phases.
Re: GitHub having issues today
#27The fact that Github has been so unstable for so long is absolutely insane to me. I know ops is hard, but this level of consistent outage points to an endemic problem. Is it the legacy rails/mysql stack that is the largest culprit or is there systemic rot in the engineering org?
Why people think it can be related to Rails when there are tons of companies out there using Rails not affected by this degradation?
One difference is that Rails and MySQL on a Github scale is rare, even when taking into account Github scale is rare.
Re: GitHub having issues today
#28The fact that Github has been so unstable for so long is absolutely insane to me. I know ops is hard, but this level of consistent outage points to an endemic problem. Is it the legacy rails/mysql stack that is the largest culprit or is there systemic rot in the engineering org?
Every time there's a GitHub outage of any severity one of the top comments on HN is inevitably suggesting that it's probably due to Rails. It's getting pretty tiresome.
Calling it a "legacy rails" stack is incredibly disingenuous as well. It's not like they're running a 5 year old unsupported version of Rails/MySQL. GitHub runs from the Rails main branch - the latest stable version they possibly can - and they update several times per month.[^1] They're one of the largest known Rails code bases and contributors to the framework. Outside of maybe 37 Signals and Shopify they employ more experts in the framework and Ruby itself than any other company.
It's far more likely the issue is elsewhere in their stack. Despite running a rails monolith, GitHub is still a complex distributed system with many moving parts.
I feel like it's usually configuration changes and infra/platform issues, not code changes, that cause most outages these days. We're all A/B testing, canary deployments, and using feature flags to test actual code changes...
[^1]: https://github.blog/2023-04-06-building-github-with-ruby-and...
Re: GitHub having issues today
#29Earlier quoted context omitted.
I'm pretty sure (hope?) the comment was said _in jest_ (though I'm not familiar with _this internet standard_)
Yeah. I thought markdown would work here.