The usual services (actions) again down around the same time. This is embarrassing.
GitHub incident 2022-03-23
71–80 of 206 posts
Re: GitHub incident 2022-03-23
#72Earlier quoted context omitted.
I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…
At my organization it's always been true. Setting up GitLab is fairly easy, in my company we do it and it's cheap (on-prem hosting is basically zero, and we had the IPs/domains already) and it hasn't given us too many headaches. I think last time I had to do something was maybe a few months ago when I restarted it so that it picked up the updated SSL certificate.
>> Setting up GitLab is fairly easy, in my company we do it and it's cheap (on-prem hosting is basically zero, and we had the IPs/domains already)
In what tech company is hosting or domains the main cost centre? Many companies spend more on a single hour of a dev's time than their entire GH monthly bill.
Re: GitHub incident 2022-03-23
#73Earlier quoted context omitted.
If you want companies to be honest on their status pages (I do!), you can't just count incidents like that. Status pages can be an amazing place to communicate all kinds of problems. Most issues have a relatively narrow impact, but the impacted people _still_ benefit from seeing them listed.
How can we solve this as customers, or push the vendor to do better?
Stop being a customer of crappy vendors
Re: GitHub incident 2022-03-23
#74https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.
> At this rate the benefits of running your own gitea or gitlab are starting to become competitive
When you host things yourself, you still have downtime. And, having worked with Github for over a decade, the actual disruption to my work is from downtime is much less than if I had to host my own.
That being said: I briefly worked for a company that hosted its own source code control system. For us, as a small team, it wasn't worth it. The system was outdated and hosted in an insecure manner. No one ever did any "admin" work except the founder. He ran it because he had irrational fears of switching, not because of any tangible advantages over Github (and competitors.)
Keep in mind that Github (and competitors) are often cheaper than the time needed to invest in hosting your own. (Estimate 10-20 hours a year of invested time. Calculate your hourly rate. Github and competitors are cheaper.) In order to come ahead, you need tangible benefits other than "I think I can have less downtime."
Re: GitHub incident 2022-03-23
#75https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.
Does Gitea support some kind of federation / cross-instance PRs? That's the main thing I'd miss from a self-hosted instance, the ease of getting contributions. After all, you don't even need Gitea for pure Git hosting. If you have a server with SSH access, just init a bare repo in a directory, push to that, and you're ready to go. No web UI needed. The reason I'm still using GitHub is not code hosting. It's collabora…
But by using GitHub for "collaboration" you are sacrificing decentralization.
Re: GitHub incident 2022-03-23
#76I really wish they would add the word “outage” to these titles. “Incident” alone makes me think something got hacked or leaked.
That's SRE lingo --> https://sre.google/sre-book/managing-incidents/
Re: GitHub incident 2022-03-23
#77https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.
Is it really though? Are engineers committing so frequently that they can't make it through a few hours without Github?
It depends on how many engineers you have! But also, there are plenty of other functions in GH besides raw git, like Wiki/PR/Issues/test/deploy pipelines, etc. It can become pretty critical.
Re: GitHub incident 2022-03-23
#78https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.
Is it really though? Are engineers committing so frequently that they can't make it through a few hours without Github?
An outage of a few hours can tank a release deadline for me, so yes.
Re: GitHub incident 2022-03-23
#79I hope they figure out what’s going on every morning. Heard from inside they don’t know why the db dies everyday but restarting it fixes it.
IIS Server had/has a memory leak in worker threads that many years ago always forced us to restart the server every few days. Starting in 6.0, they added worker thread recycling and made it a mandatory to choose a time period for every thread to be recycled. Why fix the error when you can just restart the service?
Re: GitHub incident 2022-03-23
#80Earlier quoted context omitted.
I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…
Also, looking at this it seems like GitHub isn't doing the common SaaS thing of just lying on their status page. Many providers, both internal and external, would look a lot worse if they had honest status pages.
They are green for good 15 minutes from first moment i see problems, not the first time, it happens actually quite often. Maybe that's the time they need to confirm/cross check/write status update, don't know.