Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practices? Or is it more likely that there's more to the story than what they're sharing now?
Update on 1/28 service outage
11–20 of 193 posts
Re: Update on 1/28 service outage
#12Github doesn't deploy their services in multiple az's?
Re: Update on 1/28 service outage
#13You can see the cascade effect on their status page graphs: https://status.github.com/
What is impressive is that with a website 2h down, they can still announce a 97% availability for the day even so the graph clearly shows the 2h of failures in the day... :-/
Re: Update on 1/28 service outage
#14Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…
I'm shocked as well. You would think they would deploy in multiple availability zones at the very least.
Re: Update on 1/28 service outage
#15It's one thing when one temporarily loses access to remote repositories for pushes. Quite bearable, because you can exchange code across your corporate network using patches and whatnot. And it's totally different when you cannot friggin build anything because package managers grab dependencies directly off of GitHub.
Re: Update on 1/28 service outage
#16Re: Update on 1/28 service outage
#17Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…
I've experienced a brief full-scale power loss at a data center before. It is unbelievable how much goes wrong. The machines had been chugging along for years, happily doing their job, but on the next boot the hard drives were suddenly corrupted, or the power supplies broken. The impacts of that power outage were felt for at least six months.
It's one of those things where, if you're not regularly cutting power to your data center, you're not building resilience to such a thing happening. So when it does, it's not pretty. :)
Re: Update on 1/28 service outage
#18This post makes it sound like Github has it's own data centers and power infrastructure structure, this is definitely news to me.. I'd presumed co-lo at best.
The last news I've heard about it was back in 2009, https://github.com/blog/493-github-is-moving-to-rackspace. But I've also heard that they have some infrastructure on site (clearly not what they were talking about).
Re: Update on 1/28 service outage
#19Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…
I am surprised at the data center. Power failure is one of the most basic parts of being N+1 for a data center. That is why they have batteries (last a few minutes) and then diesel generators (last days if needed).
Re: Update on 1/28 service outage
#20You can see the cascade effect on their status page graphs: https://status.github.com/
Interesting that their exception logging didn't get turned back on until this morning, from the looks of things.