GitHub: October 21 Incident Report • Hacker News Board

Re: GitHub: October 21 Incident Report

#71

Post by platinium » Mon, Oct 22, 2018, 4:14 PM UTC

goykasi wrote: post #69

alexeiz wrote: post #45

Earlier quoted context omitted.

The assumption that developer time is lost when Github is unavailable is wrong. The whole idea of Git being a distributed VCS that it does not require any connection to the main server (i.e. Github) to work with a local copy of the repository. If Github is down, I can still do my work locally and then push changes to Github when it's back online. The only case when I may get blocked, is when I need to fetch project d…

But you are ignoring a huge component of using github. If you only use github as a central, shared repo, sure you lost almost nothing. Their git infra seemed to be operational throughout this, but if you use the major features of github: issues, PRs, review, webhooks for CI, etc, you probably did lose out on developer time. Me and my team pushed tons of code to origin today (JST btw), but we were almost at a stand st…

Going forward, I see that the sensible solution for all small companies relying on cloud-hosted git is to always have a secondary cloud provider at all times.

Re: GitHub: October 21 Incident Report

#72

Post by [deleted] » Mon, Oct 22, 2018, 4:17 PM UTC

goykasi wrote: post #69

alexeiz wrote: post #45

Earlier quoted context omitted.

The assumption that developer time is lost when Github is unavailable is wrong. The whole idea of Git being a distributed VCS that it does not require any connection to the main server (i.e. Github) to work with a local copy of the repository. If Github is down, I can still do my work locally and then push changes to Github when it's back online. The only case when I may get blocked, is when I need to fetch project d…

But you are ignoring a huge component of using github. If you only use github as a central, shared repo, sure you lost almost nothing. Their git infra seemed to be operational throughout this, but if you use the major features of github: issues, PRs, review, webhooks for CI, etc, you probably did lose out on developer time. Me and my team pushed tons of code to origin today (JST btw), but we were almost at a stand st…

[deleted]

Re: GitHub: October 21 Incident Report

#73

Post by platinium » Mon, Oct 22, 2018, 4:20 PM UTC

fishnchips wrote: post #67

smilesmile2018 wrote: post #64

GitHub team seems to be VERY unprofessional. 15 hour outage means ~99.82% availability which is extremely bad. 9 hours ago they also told that they would fix the problem within 2 hours... still not fixed!!!

In order to objectively assess their level of professionalism, you'd need to know what happened, and what's going on in there right now. Think about it this way - can someone break things where you work right now to cause an outage of this magnitude? For all places that I've worked the answer would be "yes, of course".

It doesn't matter, really. Its a black box from a business perspective. Some users have lost faith, and some people will migrate to other solutions. Regardless of how fair or unfair the incident was, it is a fact that it was poor up-time especially for such an important cloud provider for code.

Re: GitHub: October 21 Incident Report

#74

Post by robinhood » Mon, Oct 22, 2018, 4:22 PM UTC

platinium wrote: post #68

On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's: https://stackoverflow.com/questions/11690709/can-a-project-h... Quoted: "Try adding a remote called "github" instead: $ git remote add github https://github.com/Company_Name/repository_name.git # push master to github $ git push github master # Push my-branch to github…

Every company at some point has some kind of incidents. It just happens. GitHub is most of the time rock solid and doesn’t deserve to be judged based on one major incident like this. On the contrary they need our support. Bitbucket and GitLab both have had problems of the same magnitude.

Re: GitHub: October 21 Incident Report

#75

Post by LaserToy » Mon, Oct 22, 2018, 4:22 PM UTC

This incident report tells very little. I hope they release what actually happened and how it affected their services. And how they are going to avoid it in the future. Almost any issue can be publicly described as: stuff broke because of network.

Re: GitHub: October 21 Incident Report

#76

Post by [deleted] » Mon, Oct 22, 2018, 4:24 PM UTC

LaserToy wrote: post #75

This incident report tells very little. I hope they release what actually happened and how it affected their services. And how they are going to avoid it in the future. Almost any issue can be publicly described as: stuff broke because of network.

[deleted]

Re: GitHub: October 21 Incident Report

#77

Post by platinium » Mon, Oct 22, 2018, 4:25 PM UTC

robinhood wrote: post #74

platinium wrote: post #68

On the plus side, this disastrous calamity by Github really made me try out Gitlab and in the process, I will now set-up a second remote on my repo's: https://stackoverflow.com/questions/11690709/can-a-project-h... Quoted: "Try adding a remote called "github" instead: $ git remote add github https://github.com/Company_Name/repository_name.git # push master to github $ git push github master # Push my-branch to github…

Every company at some point has some kind of incidents. It just happens. GitHub is most of the time rock solid and doesn’t deserve to be judged based on one major incident like this. On the contrary they need our support. Bitbucket and GitLab both have had problems of the same magnitude.

But a multiple origin solution seems the most sensical. We have failover for everything infra and services.. it seems we now need failover for cloud-based code. It just seems logical, especially if all 3 of the big cloud providers have big incidents.

Re: GitHub: October 21 Incident Report

#78

Post by 16bytes » Mon, Oct 22, 2018, 4:27 PM UTC

smilesmile2018 wrote: post #64

GitHub team seems to be VERY unprofessional. 15 hour outage means ~99.82% availability which is extremely bad. 9 hours ago they also told that they would fix the problem within 2 hours... still not fixed!!!

You need to judge them by the recovery in the context of what happened. GitHub has a world class engineering team, but major outages still happen. Even in the context of a major outage, they have best-in-class status pages and frequent around-the-clock updates.

I helped manage a hosted DVCS and CI system in my previous job, and do you know what we would've called an outage that happened at 7:00PM and had recovered partially before start of business?

A Tuesday.

Wait for the RCA to come out before throwing any stones.

Re: GitHub: October 21 Incident Report

#79

Post by edejong » Mon, Oct 22, 2018, 4:33 PM UTC

tjoff wrote: post #41

edejong wrote: post #39

Earlier quoted context omitted.

I don't think 2 hours is an exaggeration for an active user. The 50% might be too high, I agree.

2 hours lost? Maybe 2 hours affected, but then just don't push/fetch during that time. Very few would have to stop and wait. Average lost I'd guess to 10 minutes (most of it being "huh, wonder what's up with github")

You lose quite a bit more. People keep checking if it's back online. The usual flow is disrupted. Questions might be asked about whether branches can be merged. Integration with CI might fail. Managers start asking questions why features cannot be deployed. Some of the tooling I wrote even would stop working because the API was unavailable.

Unfortunately, it's easier to down vote, than to come with a better estimate of the total cost of a 13 hour down-time of github.

Re: GitHub: October 21 Incident Report

#80

Post by goykasi » Mon, Oct 22, 2018, 4:41 PM UTC

platinium wrote: post #71

goykasi wrote: post #69

Earlier quoted context omitted.

But you are ignoring a huge component of using github. If you only use github as a central, shared repo, sure you lost almost nothing. Their git infra seemed to be operational throughout this, but if you use the major features of github: issues, PRs, review, webhooks for CI, etc, you probably did lose out on developer time. Me and my team pushed tons of code to origin today (JST btw), but we were almost at a stand st…

Going forward, I see that the sensible solution for all small companies relying on cloud-hosted git is to always have a secondary cloud provider at all times.

> Which brings into question... what is so special about the "front-end" features that Git provides? ... bare bones skeleton that we can swap in/out at will

Your comment disappeared. I think you had a good point. And itd be cool if it could be a real thing, but...

I dont disagree. But our team realized that we rely almost too much on github; we just decided to put up with it. Is there a solution that doesnt depend on running your own "github"? My infra lead and I had the usual, fun tongue and cheek chat that began with... them: "maybe its time we switch to gitlab", me: "can we be up tomorrow?" Weve had that same discussion many times before.

In the end, it comes down to process. If you buy into the features, PRs, CI hooks, etc, then its really hard to just say "well we can maintain and replicate the alternative for the .1% edge-case". Otherwise, you might as well just use that and not github, gitlab, etc. Its hard to decouple from github. They do that by nature -- dev still continues; they are a piece of the process puzzle. I think abstracting them away just complicates things unnecessarily.