Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Update on 1/28 service outage

github.com

Post reply on HN
Print view View original

161–170 of 193 posts

Re: Update on 1/28 service outage

#161

Post by semi-extrinsic »

nemothekid wrote: post #116
seiji wrote: post #92

Earlier quoted context omitted.

Exactly right, but over the past six years there's been a strong (and accelerating) trend among developers of " lalala we don't want to know how anything works! give us an API and go away." Most developers I've seen reject even learning about networks or DNS or operating systems or databases. Such willful ignorance boggles the mind, but they are praised because their goals are shipping half-broken things as rapidly a…

> Exactly right, but over the past six years there's been a strong (and accelerating) trend among developers of "lalala we don't want to know how anything works! give us an API and go away." I will argue that this trend has always existed. I'm sure you can find an x86/68k/z80 developer complaining that developers are going "lalala we don't want to know how anything works! give us an the C-language and go away."* I'm…

The good reverend Laphroaig preaches:

If the 0day in your familiar pastures dwindles, despair not! Rather, bestir yourself to where programmers are led astray from the sacred Assembly, neither understanding what their programming languages compile to, nor asking to see how their data is stored or transmitted in the true bits of the wire. For those who follow their computation through the layers shall gain 0day and pwn, and those who say “we trust in our APIs, in our proofs, and in our memory models and need not burden ourselves with confusing engineering detail that has no scientific value anyhow” shall surely provide an abundance of 0day and pwnage sufficient for all of us.

Re: Update on 1/28 service outage

#162

Post by Pharaoh2 »

gregmac wrote: post #151
Pharaoh2 wrote: post #147

Earlier quoted context omitted.

Except that it is not. It redundant array of computers, if one goes down, another takes it place and all the apps running on it are migrated to the new hardware. And if the whole zone goes down, the apps are migrated to a different zone. If the whole region goes down, the apps can be migrated to a different region. The 9s are so high that you don't have to worry about hardware issues anymore, unlike when you are runn…

Or it could be literally an old desktop computer sitting in someone's damp basement on a DSL connection. The problem with just saying "the cloud" is you can't tell the difference.

Generally when people say the cloud, they mean one of the big Public/Private cloud providers, not someone's basement.

Re: Update on 1/28 service outage

#163

Post by JimDabell »

JetSetWilly wrote: post #48
LinuxBender wrote: post #27

Earlier quoted context omitted.

I am not at all surprised. There are 'best practices' and then there is what really happens based on business processes and needs. In reality, even the most cloudy of cloud providers will run into this problem at some point. Folks often come up with ideas of implementing something like Chaos Monkey in their data-center, then realize the actual impact it will have and find it is almost impossible to get the rest of th…

The problem is most environments are very heteregenous. I evaluated chaos monkey approach for a big bank, the issue is that netflix has whole data centres full of loads of machines doing pretty much the same thing, streaming and serving. And the worst that can happen is a customer's stream stops and they have to restart it. But in most big companies you have thousands of apps that are all doing very different things.…

> Perhaps a critical app might run on 4 hosts spread across two data centres - you're not going to convince people to have chaos monkey regularly and randomly bringing down these hosts, it would cause real impact and is risky. Yeh in theory it should be able to cope but in reality the scales in most orgs are quite different.

The difference between theory and reality is precisely the reason Chaos Monkey and tools like it exist.

What you're essentially saying is that in theory, these systems have been designed to be resilient, but in reality, they may not be. If that's the case, then you'd better verify your resiliency, because being resilient in theory but not reality isn't going to help you when your service goes down.

Re: Update on 1/28 service outage

#164

Post by snowwrestler »

nickpsecurity wrote: post #136
nemothekid wrote: post #130

Earlier quoted context omitted.

There is no way to configure mirrors with the project-specific tooling (AFAIK). Git, (and like most other DVCS) supports mirroring. For example Linux, hosted on Github, ( https://github.com/torvalds/linux/commits/master ) is also mirrored and hosted on kernel.org ( https://git.kernel.org/cgit/ ). Or, the apache projects ( https://github.com/apache/cassandra ), which are also hosted on apache.org ( https://git-wip-us.…

So, the protocol is definitely good enough to handle situations like this but just commonly applied that way esp with many Github-hosted projects. Gotcha. That makes sense.

Git is very flexible and does not even require repo-to-repo communication over the wire at all; patches can be emailed among contributors and then committed and tracked locally. Branching and merging is so fast and easy in git that every participant can have a slightly different repo for a given project, incorporating shared changes as they see fit.

Github is popular because it is opinionated--it chooses to use git in certain ways, thus reducing the complexity for people who aren't git experts (i.e. most people).

The most sophisticated users of git--the Linux and git projects, probably--do not rely on github at all. As far as I know, they share code via emailed patches. Some of those developers might not even be using git at all! They just send patches upstream, and the upstream developer checks the patches into their local git repo and then preps a larger patch to be emailed farther upstream.

Re: Update on 1/28 service outage

#165

Post by nickpsecurity »

debaserab2 wrote: post #160
nickpsecurity wrote: post #72

Earlier quoted context omitted.

Hence Git/Github in my comment. I already know there's a difference. I just don't know much more than that until I learn the two.

If you understand the difference between the two, you'd realize your comment makes no sense. The fact that github went down due to a power failure has nothing to do with git as a solution. The fact that everyone uses git more or less the same as svn is the problem. Git is decentralized, but because so many people rely on github most don't ever use the decentralized aspect to it.

If you understood my comment, you'd know I don't understand the differences between the two that much since I haven't studied them yet. Been clear in a few comments on that. The reason I associate them here is that most projects I see don't just use Git: they use Github, too. So, I briefly wonder and get feedback about how inherent Github-style downtime was or if it was configuration/deployment issues.

Several commenters helpfully described how Git can easily prevent stuff like this and that project-level stuff is why this is a liability. That's good to know as it's already a selling point to management types for a solution like it. Can just ensure the problem doesn't show up in a local deployment by a wiser configuration.

Re: Update on 1/28 service outage

#166

Post by bkeroack »

Pharaoh2 wrote: post #147
Zelmor wrote: post #94

Earlier quoted context omitted.

One has to keep in mind that there is no cloud. It's just someone else's computer.

Except that it is not. It redundant array of computers, if one goes down, another takes it place and all the apps running on it are migrated to the new hardware. And if the whole zone goes down, the apps are migrated to a different zone. If the whole region goes down, the apps can be migrated to a different region. The 9s are so high that you don't have to worry about hardware issues anymore, unlike when you are runn…

That's the theory (or the marketing pitch, depending upon perspective).

The reality can be rather different[1][2][3].

1. http://money.cnn.com/2011/04/21/technology/amazon_server_out...

2. http://www.zdnet.com/article/amazon-web-services-suffers-out...

3. http://www.theregister.co.uk/2015/09/20/aws_database_outage/

Re: Update on 1/28 service outage

#167

Post by gaius »

cdelsolar wrote: post #156
forrestthewoods wrote: post #149

Earlier quoted context omitted.

So your proposed solution is one of the only two hard problems in computer science? That should be a solid clue that you're wrong. "There are only two hard things in Computer Science: cache invalidation and naming things." -- Phil Karlton

wow, i never realized cache invalidation was one of the ONLY two hard problems in CS

The quote is supposed to be, two hard problems: cache invalidation, naming conventions and off-by-one errors.

Re: Update on 1/28 service outage

#168

Post by nickpsecurity »

snowwrestler wrote: post #164
nickpsecurity wrote: post #136

Earlier quoted context omitted.

So, the protocol is definitely good enough to handle situations like this but just commonly applied that way esp with many Github-hosted projects. Gotcha. That makes sense.

Git is very flexible and does not even require repo-to-repo communication over the wire at all; patches can be emailed among contributors and then committed and tracked locally. Branching and merging is so fast and easy in git that every participant can have a slightly different repo for a given project, incorporating shared changes as they see fit. Github is popular because it is opinionated--it chooses to use git i…

That's pretty wild. Sounds like main program/protocol is very true to the UNIX philosophy of tooling. My early reads on it suggested that gave it both it's power/versatility and horrific UI consequences for beginners. An opinionated UI and host like Github is a natural consequence.

I remember thinking in my early reading that git was like an assembly language for build systems. It really needed a front-end of some kind to smooth things over for new and casual users. Maybe not as heavyweight as Github but better than the main program. Can keep the low-level stuff in for advanced users.

Was that or is that still a common assessment or was my initial impression off?

Re: Update on 1/28 service outage

#169

Post by JetSetWilly »

JimDabell wrote: post #163
JetSetWilly wrote: post #48

Earlier quoted context omitted.

The problem is most environments are very heteregenous. I evaluated chaos monkey approach for a big bank, the issue is that netflix has whole data centres full of loads of machines doing pretty much the same thing, streaming and serving. And the worst that can happen is a customer's stream stops and they have to restart it. But in most big companies you have thousands of apps that are all doing very different things.…

> Perhaps a critical app might run on 4 hosts spread across two data centres - you're not going to convince people to have chaos monkey regularly and randomly bringing down these hosts, it would cause real impact and is risky. Yeh in theory it should be able to cope but in reality the scales in most orgs are quite different. The difference between theory and reality is precisely the reason Chaos Monkey and tools like…

That's true, but if an app, say, is running on 4 hosts doing some boutique thing for a small unit of 20 traders, then the practical reality is that they might not want Chaos Monkey bringing down 25% of the throughput randomly, and interrupting whatever actual cash money requests are in progress on a host.

Itsa lot easier to promote that if it is thousands of servers doing something fairly mundane where, worst-case, it not working means a tiny tiny proportion of your customers have to restart their video stream. So what?

But for a small hetereogenous business where what's happening has a much higher cash density, the actual practicalities of randomly killing things in production and the risk that represents rather get in the way, even though in theory you should be able to kill anything in production with minimal impact, you are much less inclined to take that risk when the stakes are higher.

Re: Update on 1/28 service outage

#170

Post by nickpsecurity »

dsjoerg wrote: post #82
forrestthewoods wrote: post #42

Earlier quoted context omitted.

And people give me shit when I argue that open source projects should include 100% of dependencies.

how far down the stack do you go? do open source projects need to include their own compiler? what would compile it?

I suggested how far they need to go in context of Debian's reproducible builds posts:

https://news.ycombinator.com/item?id=10182282

That would solve readability, plenty of subversion, verifiability, much of portability, and perform anywhere from OK to good. Not going to happen but academics and proprietary software already did it to varying degrees. As post noted, traceability & verification from requirements to specs to code to object code is a requirement for high assurance systems. My methods, mostly borrowed from better researchers, are the easiest ones to use.

Post reply on HN