Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub incident 2022-03-23

githubstatus.com

Post reply on HN
Print view View original

151–160 of 206 posts

Re: GitHub incident 2022-03-23

#151

Post by belter »

mbesto wrote: post #136
belter wrote: post #39

Earlier quoted context omitted.

Excluding ones reported as [Errors], [Scheduled] or [Notifications] 2019 -> 39 Incidents 2020 -> 67 Incidents 2021 -> 86 Incidents 2022 -> 20 Incidents so far Edit: Using Linear Regression...Prediction for total end 2022: 111 Incidents.

The number of incidents isn't so much of a problem as the amount of downtime is. That would be more interesting to see.

GitHub Availability Report [1]

Service Downtime Core Services Only - Cumulative per Month

( Some months with more than one outage)

Jan 2021: 3 hours 53 min

Feb 2021: 1 hour 42 min

Mar 2021: 4 hours 10 min

Apr 2021: 2 hours 20 min

May 2021: 10 hours 34 min

Jun 2021: 0 min

Jul 2021: 0 min

Aug 2021: 4 hours 23 min

Sep 2021: 0 min

Oct 2021: 1 hour 36 min

Nov 2021: 2 hours 50 min

Dec 2021: 0 min

Jan 2022: 26 min

Feb 2022: 13 min

[1] https://github.blog/tag/github-availability-report/

Re: GitHub incident 2022-03-23

#152

Post by jhugo »

edgyquant wrote: post #22
nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…

I think the impact was for some reason not consistent between users (maybe due to geographical factors or maybe sharding of accounts?). We're in Asia and I think we've had three different days recently where we couldn't actually get much work done due to GitHub being flakey or down for the entire day and our CI/CD and development processes being built around it. We ended up moving off GitHub onto a self-hosted system, which took about a day of work for one engineer (CI/CD itself was already self-hosted, so just Git, issues and PRs), and there have already been two more GitHub outages since then.

Re: GitHub incident 2022-03-23

#153

Post by jhugo »

thaeli wrote: post #60
edgyquant wrote: post #22

Earlier quoted context omitted.

I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…

Also, looking at this it seems like GitHub isn't doing the common SaaS thing of just lying on their status page. Many providers, both internal and external, would look a lot worse if they had honest status pages.

Several of the recent outages were much longer (at least for us, here in Asia) than they admitted on their status page. In one case I started work, noticed I couldn't push to or pull from GitHub, that situation persisted all day, and around 5pm local time (so morning-ish in the US) suddenly their status page acknowledged the problem and a discussion started on HN.

Re: GitHub incident 2022-03-23

#154

Post by sitzkrieg »

zrail wrote: post #138
zeku wrote: post #118

Earlier quoted context omitted.

Just a bone to pick... new grad engineers in my US city started around 60-70k in 2018 when my college cohort graduated. Southern US...

Things have changed considerably over the last four years.

yea and starting salary for zero experience developers is not $120k in most places

Re: GitHub incident 2022-03-23

#155

Post by sitzkrieg »

TheRealPomax wrote: post #102
nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

But how many of those actually affected you? For example, no amount of issues around codespaces or github packages would impact my professional use of github, so whether there are 21 or 5000 or those parts get permanently taken offline makes no difference in what I need out of the platform. How many core incidents? The part that affects whether you can even push to and pull from a repo, and access issues and PRs? Bec…

i could not even sso login so it was a bit more impactful than it sounds on the paper

Re: GitHub incident 2022-03-23

#156

Post by jeltz »

edgyquant wrote: post #48
jeltz wrote: post #27

Earlier quoted context omitted.

Maybe you are in a different time zone because our organization certainly noticed and was disrupted by this.

I’m on PST time, some of our other devs are on the east coast and one is in India. I think we’re spread out enough it should be an issue but maybe we prioritize different things.

We are in CET and maybe we use Github differently than you.

Re: GitHub incident 2022-03-23

#157

Post by jeltz »

TheRealPomax wrote: post #102
nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

But how many of those actually affected you? For example, no amount of issues around codespaces or github packages would impact my professional use of github, so whether there are 21 or 5000 or those parts get permanently taken offline makes no difference in what I need out of the platform. How many core incidents? The part that affects whether you can even push to and pull from a repo, and access issues and PRs? Bec…

I was affected by the one last week, the one yesterday and the one today. The one today was harmless but the other two disrupted out work. All three were "core incidents", but the one today felt shorter.

Re: GitHub incident 2022-03-23

#158

Post by jacobr »

edgyquant wrote: post #22
nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…

20 PRs waiting in line for half a day to be merged is pretty annoying. We’ve had that on multiple occasions the last few weeks due to GitHub incidents.

Re: GitHub incident 2022-03-23

#159

Post by dijonman2 »

nimbius wrote: post #16

https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.

GitHub enterprise is amazing, but I agree that a centrally hosted Git instance of any variety is a liability.

With the advent of the Okta breach I think we will see a reverse in the centralization trend.

Re: GitHub incident 2022-03-23

#160

Post by zenexer »

wnevets wrote: post #150
edgyquant wrote: post #22

Earlier quoted context omitted.

I’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold.…

> I’ve hardly noticed the three month surge. This has been my experience as well. I don't know if that means GitHub is being overly transparent about issues or I've just been lucky but I would hate if people punished services for being transparent and informative on their status pages.

GitHub's outages have hit me hard over the past week or so. I don't think it's a matter of them being transparent--if anything, I was hitting errors well before their status page updated. Yesterday it was completely unusable for much of my workday, and today tasks that normally take me a few minutes have been taking hours.
Post reply on HN