Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update on 1/28 service outage

github.com

Post reply on HN
Print view View original

21–30 of 193 posts

Re: Update on 1/28 service outage

#21

Post by arthurschreiber »

Loic wrote: post #13
frik wrote: post #3

You can see the cascade effect on their status page graphs: https://status.github.com/

What is impressive is that with a website 2h down, they can still announce a 97% availability for the day even so the graph clearly shows the 2h of failures in the day... :-/

Unless I'm mistaken, 97% of (24 hours) = 23.28 hours.

Re: Update on 1/28 service outage

#22

Post by vorador »

skewart wrote: post #11

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…

I have no doubt the people at Github have spent a lot of time thinking about multi-region failover. You never hear about the successful failovers --- only the ones which cause outages. To quote a famous US politician: "There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don't know we don't know."

You can't failover things you didn't predict.

Re: Update on 1/28 service outage

#23

Post by cookiecaper »

bhaak wrote: post #10

"Millions of people and businesses depend on GitHub" Well, we shouldn't depend on it so much. I shudder at the thought what an outage of GitHub would mean for our company. This time, we were lucky as it was during the night in Europe. Unfortunately, I don't have the power to test this scenario in our company.

It shouldn't really have much effect. One of git's major selling points is that it's a DVCS, meaning that everyone has a local copy of the repository. Perhaps some collaboration features will be down for a couple of hours (which I think is a downside to GitHub's decision not to put issues/PR history inside of git), but everyone should still be able to do work, commit to the repo, review history, and so forth. If you have people who do code, they can probably find something to work on for two hours without having the Issues/PR interface, right?

Re: Update on 1/28 service outage

#24

Post by coliveira »

skewart wrote: post #11

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…

My experience matches exactly what Github says. Power outages can bring down even the best systems. The problem is that it is never clear what parts of the systems will continue to work in these situations, until it actually happens. Especially when you're talking about complex applications that depend on many moving pieces. The point is, the more complex your online app, the more points of failure can be exposed in these situations.

Re: Update on 1/28 service outage

#26

Post by rch »

Zikes wrote: post #7
[deleted] wrote: post #4

[deleted]

I'm starting to think that people should mirror their packages to BitBucket as a rule, and that package managers should round robin/flip a coin between the two, or use whichever is available in case of outages.

I'd rather have something like Netflix's Open Connect Appliances, covering all of Github, sitting in each office and a centrally located colo facility.

Re: Update on 1/28 service outage

#27

Post by LinuxBender »

skewart wrote: post #11

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…

I am not at all surprised. There are 'best practices' and then there is what really happens based on business processes and needs. In reality, even the most cloudy of cloud providers will run into this problem at some point. Folks often come up with ideas of implementing something like Chaos Monkey in their data-center, then realize the actual impact it will have and find it is almost impossible to get the rest of the business to agree to this concept. It isn't as easy at it sounds. I only know of two businesses that have actually implemented Chaos Monkey; one being the company that coined the term. Even regular reboots won't catch these problems and if folks were honest, you would see +1 year up-times on most servers in most places. That is just based on my experiences and I am sure some of you have seen different.

Re: Update on 1/28 service outage

#28

Post by raverbashing »

bhaak wrote: post #10

"Millions of people and businesses depend on GitHub" Well, we shouldn't depend on it so much. I shudder at the thought what an outage of GitHub would mean for our company. This time, we were lucky as it was during the night in Europe. Unfortunately, I don't have the power to test this scenario in our company.

Have a local repos that mirrors the master one on GitHub periodically

Should that fail, start working on the local repos until github is back, then sync back to it

Re: Update on 1/28 service outage

#29

Post by [deleted] »

Loic wrote: post #13
frik wrote: post #3

You can see the cascade effect on their status page graphs: https://status.github.com/

What is impressive is that with a website 2h down, they can still announce a 97% availability for the day even so the graph clearly shows the 2h of failures in the day... :-/

[deleted]

Re: Update on 1/28 service outage

#30

Post by brink »

skewart wrote: post #11

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…

Gotta love it when a top comment starts with "Am I the only one".
Post reply on HN