Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update on 1/28 service outage

github.com

Post reply on HN
Print view View original

31–40 of 193 posts

Re: Update on 1/28 service outage

#31

Post by pc86 »

raverbashing wrote: post #28
bhaak wrote: post #10

"Millions of people and businesses depend on GitHub" Well, we shouldn't depend on it so much. I shudder at the thought what an outage of GitHub would mean for our company. This time, we were lucky as it was during the night in Europe. Unfortunately, I don't have the power to test this scenario in our company.

Have a local repos that mirrors the master one on GitHub periodically Should that fail, start working on the local repos until github is back, then sync back to it

Depending on your definition of "periodically" you may lose almost as much time to syncing back than the outage would have caused without the local mirrors.

Re: Update on 1/28 service outage

#32

Post by [deleted] »

skewart wrote: post #11

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…

[deleted]

Re: Update on 1/28 service outage

#33

Post by detaro »

skewart wrote: post #11

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…

Stuff happens, and even if you test all kinds of things real failure situations always can work differently, with partial failures etc. Just takes one important subsystem hitting an unforeseen edge case, and going completely down is in many cases better than risking running in a state that destroys data or does other bad things. Same for taking your time to go back online.

The cases that work are not the ones you hear about. Best practices and testing reduce the risk of making the news, but can't guarantee success.

Re: Update on 1/28 service outage

#34

Post by WillAbides »

Loic wrote: post #13
frik wrote: post #3

You can see the cascade effect on their status page graphs: https://status.github.com/

What is impressive is that with a website 2h down, they can still announce a 97% availability for the day even so the graph clearly shows the 2h of failures in the day... :-/

The 97% you see on the status page is for the past 24 hours. That doesn't include any of the outage being discussed here.

Re: Update on 1/28 service outage

#36

Post by mirekrusin »

arthurschreiber wrote: post #21
Loic wrote: post #13

Earlier quoted context omitted.

What is impressive is that with a website 2h down, they can still announce a 97% availability for the day even so the graph clearly shows the 2h of failures in the day... :-/

Unless I'm mistaken, 97% of (24 hours) = 23.28 hours.

yes, it went down to 89% or something just after the problem.

Re: Update on 1/28 service outage

#37

Post by rburhum »

Yesterday I was being a bit of an ass to a few people about how "the whole point of using git is so that we can do decentralized code management and why these dependencies were being pulled from our private github if the could be sent point to point yadda yadda yadda". Then they proceeded to go over the list of package managers and dependencies we used and I had to shut up. Even when we host our own Docker Hub and package managers (we do), if you dig far enough, you can find some dependency of a dependency of dependency that relies on GitHub. Brew/npm/build script/whatever. It is crazy how everything has changed so much in the past few years. GitHub went from something that was really nice to have to a core requirement for complex systems that rely heavily on open source.

Re: Update on 1/28 service outage

#38

Post by wyldfire »

skewart wrote: post #11

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…

The only way you can build fault-resilient systems is to frequently test fault injection scenarios. Netflix is pretty mature in this regard, perhaps Github can learn from their example.

That said, it's possible that github may have considered that this particular style of outage is rare enough that they don't want to make their design tolerate it. Though if that were the case, I'd wager they'd re-evaluate the cost/benefit right around now. :)

Re: Update on 1/28 service outage

#39

Post by untog »

vorador wrote: post #22
skewart wrote: post #11

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…

I have no doubt the people at Github have spent a lot of time thinking about multi-region failover. You never hear about the successful failovers --- only the ones which cause outages. To quote a famous US politician: "There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the o…

Heh, that quote always amuses me. People hated it, but it actually does make a lot of sense.
Post reply on HN