Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update on 1/28 service outage

github.com

Post reply on HN
Print view View original

71–80 of 193 posts

Re: Update on 1/28 service outage

#71

Post by seiji »

tommoor wrote: post #6

This post makes it sound like Github has it's own data centers and power infrastructure structure, this is definitely news to me.. I'd presumed co-lo at best.

"data center" is a confusing term.

Very few companies build their "data centers" (apple, google, amazon, NSA, actual 'data center' companies, etc). Most companies rent cage space in a larger data center and call that their "private data center." Smaller companies will rent a few dedicated servers or colo half racks from other resellers.

Re: Update on 1/28 service outage

#72

Post by nickpsecurity »

debaserab2 wrote: post #65
nickpsecurity wrote: post #59

Earlier quoted context omitted.

That's a good point. I've been ignoring learning Git as long as I can but almost everything on my todo list heavily uses it. Or ties into it as you said. So, I'm going to have to bite the bullet and learn it. Yet, I swore Git fans told me its decentralized design avoids single points of failures where everyone has a copy and can still work when a node is down just not necessarily coordinate or sync in a straight-forw…

git != github

Hence Git/Github in my comment. I already know there's a difference. I just don't know much more than that until I learn the two.

Re: Update on 1/28 service outage

#73

Post by drather19 »

JetSetWilly wrote: post #48
LinuxBender wrote: post #27

Earlier quoted context omitted.

I am not at all surprised. There are 'best practices' and then there is what really happens based on business processes and needs. In reality, even the most cloudy of cloud providers will run into this problem at some point. Folks often come up with ideas of implementing something like Chaos Monkey in their data-center, then realize the actual impact it will have and find it is almost impossible to get the rest of th…

The problem is most environments are very heteregenous. I evaluated chaos monkey approach for a big bank, the issue is that netflix has whole data centres full of loads of machines doing pretty much the same thing, streaming and serving. And the worst that can happen is a customer's stream stops and they have to restart it. But in most big companies you have thousands of apps that are all doing very different things.…

While Netflix as a company is focused at doing one specific thing at large scale, they're heavily vested in microservices and do actually have "thousands of apps that are all doing very different things".

Chaos Monkey fits when people build and deploy their services with the notion that any particular instance (or dependency) could fail at any given time. It's a tough road to evolve out of a legacy, monolithic stack without much redundancy baked in.

Re: Update on 1/28 service outage

#74

Post by lomnakkus »

LinuxBender wrote: post #27
skewart wrote: post #11

Am I the only one who is a little shocked that a power outage could have such a huge effect and bring them down for so long? I'm not an infrastructure guy, and I don't know anything about Github's systems, but aren't data center power outages pretty much exactly the kind of thing you plan for with multi-region failover and whatnot. Is it actually frighteningly easy for kind of to happen despite following best practic…

I am not at all surprised. There are 'best practices' and then there is what really happens based on business processes and needs. In reality, even the most cloudy of cloud providers will run into this problem at some point. Folks often come up with ideas of implementing something like Chaos Monkey in their data-center, then realize the actual impact it will have and find it is almost impossible to get the rest of th…

> In reality, even the most cloudy of cloud providers will run into this problem at some point.

Actually, wasn't this[0] what did happen several years ago when Amazon Ireland went down for days on end?[1]

[0] TL;DR: Cascading effects of power outage.

[1] http://readwrite.com/2011/08/08/amazons-ireland-services-sti... (didn't read the article, it was just high in the google search results)

Re: Update on 1/28 service outage

#75

Post by seiji »

rburhum wrote: post #37

Yesterday I was being a bit of an ass to a few people about how "the whole point of using git is so that we can do decentralized code management and why these dependencies were being pulled from our private github if the could be sent point to point yadda yadda yadda". Then they proceeded to go over the list of package managers and dependencies we used and I had to shut up. Even when we host our own Docker Hub and pa…

if you dig far enough, you can find some dependency of a dependency of dependency that relies on GitHub. Brew/npm/build script/whatever

But really, why?

Is it just institutional laziness on the part of all developers? We had reliable rsync CPAN mirrors in 1995. In the early days of the Internet, companies would mutually host secondary DNS for each other to be more reliable. For some reason, we've forgotten all about reliability and disaster recovery and geographical distribution. Now the collective programmer mindset with regards to global infrastructure seems to be "lol, we're too dumb to make things work, let's just outsource everything to closed source, for-profit companies and hope for the best."

Re: Update on 1/28 service outage

#76

Post by nickpsecurity »

ajkjk wrote: post #69
nickpsecurity wrote: post #59

Earlier quoted context omitted.

That's a good point. I've been ignoring learning Git as long as I can but almost everything on my todo list heavily uses it. Or ties into it as you said. So, I'm going to have to bite the bullet and learn it. Yet, I swore Git fans told me its decentralized design avoids single points of failures where everyone has a copy and can still work when a node is down just not necessarily coordinate or sync in a straight-forw…

I'm sorry to be rude, but, it sounds like you should go learn Git and come back to this conversation. The decentralized design does avoid single points of failures, and everyone does have a copy. So - check, check, great. Unfortunately (maybe..) everyone has put their master repos in the same place, which somewhat counteracts the decentralization. But there is certainly no immediate coupling between the Git repositor…

"I'm sorry to be rude, but, it sounds like you should go learn Git and come back to this conversation."

It looks like it.

"The decentralized design does avoid single points of failures, and everyone does have a copy. "

So, like many decentralized systems I've used, a master node gets worked around by other nodes who communicate in another way? Or would some retarded situation be possible where...

"Unfortunately (maybe..) everyone has put their master repos in the same place, which somewhat counteracts the decentralization."

...one node going down could prevent collaboration? Oh, you answered that. That sounds better than CVS but shit by distributed systems standards. I'll still learn it anyway since everyone is using it. Probably in next week or two.

Re: Update on 1/28 service outage

#77

Post by paultela »

why-el wrote: post #44
chadaustin wrote: post #17

Earlier quoted context omitted.

I've experienced a brief full-scale power loss at a data center before. It is unbelievable how much goes wrong. The machines had been chugging along for years, happily doing their job, but on the next boot the hard drives were suddenly corrupted, or the power supplies broken. The impacts of that power outage were felt for at least six months. It's one of those things where, if you're not regularly cutting power to yo…

> if you're not regularly cutting power to your data center, you're not building resilience to such a thing happening Would love to read examples on who is doing this and how? Reminds me of Netflix's Choas monkey, only applied to electricity. :p

There's a mention of Facebook regularly doing this in the summary section of this instagram engineering post: http://engineering.instagram.com/posts/548723638608102/

EDIT: Here's more info: http://www.datacenterknowledge.com/archives/2014/09/15/faceb...

Re: Update on 1/28 service outage

#78

Post by gavazzy »

Would it be possible for a cross between Git and Torrents? Rather than having a central server to pull/push from, instead the server would provide a list of clients. If the server goes down, the list is still available, and so people who depend on it would be able to communicate.

Re: Update on 1/28 service outage

#79

Post by ljk »

Maybe I'm ignorant, but why do companies rely on github? Why not just host it in-house? If there's power outage in the office then everything would be down anyways, right?

Re: Update on 1/28 service outage

#80

Post by nickpsecurity »

jacquesm wrote: post #64
nickpsecurity wrote: post #53

Earlier quoted context omitted.

We've been mitigating against this kind of thing with backups at other datacenters or colos for a while. They can be hot standby, cold standby, slightly degraded in performance, whatever. I also recommend the backup be on a different part of the overall power grid in case it cascades in failure. The good colo's often have connections to multiple backbones, too, which is extra redundancy. That all assumes there's a to…

I doubt it matters to anybody but was it really necessary to kill the fish?

Dude, I was thinking the same thing! That was seriously f*ed up. They should've left some cool fireworks or something left-over from July 4th. Or some safe-ish chemical that would make colorful smoke. All kinds of tricks you can do without killing live animals.

I mean, I've heard about things so wrong and ease it's like shooting fish in a bucket but... exploding fish in a datacenter? That's on another level.

Post reply on HN