Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

251–260 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#251

Post by contingencies »

rrdharan wrote: post #243
halayli wrote: post #218

This isn't the first time a config system at Google causes a major outage. https://googleblog.blogspot.com/2014/01/todays-outage-for-se...

That's entirely unsurprising. The recent major Facebook outage was also caused by bad configuration IIRC. See: http://danluu.com/postmortem-lessons/ > Configuration > > Configuration bugs, not code bugs, are the most common cause > I’ve seen of really bad outages. When I looked at publicly available > postmortems, searching for “global outage postmortem” returned > about 50% outages caused by configuration changes. P…

Great link there! Also check out his list of public postmortems at https://github.com/danluu/post-mortems

PS. On HN you should use asterisks to italicize instead of > for quoting.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#252

Post by harryf »

ben_jones wrote: post #64
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

> The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving.

Most people have a greater fear of flying than driving by car although statistically you're far more at risk in a car. One cause of that fear of flying is loss of control; you have to accept placing your life in someone else's hands.

With self driving cars suspect lack of control will also be a problem. Either we need to provide passengers with some vestige of control to keep them busy or we just wait a generation until people get used to it

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#253

Post by nxzero »

Gravityloss wrote: post #237
nxzero wrote: post #214

Earlier quoted context omitted.

Shit happens, when it comes to engineering, I'd trust Google more than even likely Boeing to manage systemic risk. As for cars, it's a real risk, but not the same as the bugs Google experienced; I personally have experienced a "bug" driving a car at high speeds, which resulted in a number of major electronic systems failing due to custom systems installed by a well known US startup.

Depends. You design and operate for a certain "shit happens" probability and price. That's why I brought up airliners. You can't set low reliability goals and just say "shit happens". You would have less buyers, and it's not even legal anymore. So the bullet was bit and more reliable aircraft were developed. In the software world we're more like nineteen twenties still. That changed. https://en.wikipedia.org/wiki/TWA…

As it relates to airplanes, "shit happens" still applies. I was flying into NYC one time and air traffic control mistakenly allowed the plane I was on to attempt a landing while another plane was taking off; my pilot don't even notice the other plane until we were over the runway. Later found out NYC depends in a number of cases for pilots to avoid collision by literally looking out the window for traffic in their fight path.

>> "I see that there could be some business value in more reliable cloud platforms."

Likely, though I have no idea how much Google is making with cloud services, but Amazon I believe is making tens of billions alone with its cloud services. That said, Amazon as far as I'm able to recall has had far worse issues and appears to be doing fine as a business.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#254

Post by stakecounter »

Bjartr wrote: post #235
pyre wrote: post #201

Earlier quoted context omitted.

Well, this bug took down the entire system. What happens when self-driving software hits a similar bug? I don't think that there is any precedent for that sort of thing with manually driven cars. The scale could easily be larger than 100-car pile-ups due to poor weather conditions.

>I don't think that there is any precedent for that sort of thing with manually driven cars. A stroke or heart attack while driving?

That would only affect the person who had a stroke or heart attack and their close vicinity. The parent is suggesting a scenario where the manual driving equivalent would be every single driver getting a stroke or heart attack at the same time.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#255

Post by nothrabannosir »

harryf wrote: post #252
ben_jones wrote: post #64

Earlier quoted context omitted.

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

> The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. Most people have a greater fear of flying than driving by car although statistically you're far more at risk in a car. One cause of that fear of flying is loss of control; you have to accept placing your life in someone else's hands. With self driving cars suspect lack of control will als…

> Most people have a greater fear of flying than driving by car although statistically you're far more at risk in a car. One cause of that fear of flying is loss of control; you have to accept placing your life in someone else's hands.

Really? That sounds counter intuitive. You'd think the reason people are afraid of flying is, because, you know, it's flying. Thirty thousand feet between you and the cold, hard ground. That's a long fall of agony to utmost certain death, and some magic turbo voodoo keeping you from it.

Would people with fear of flying really rather be the pilot?

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#256

Post by totally »

> However, in this instance a previously-unseen software bug was triggered, and instead of retaining the previous known good configuration, the management software instead removed all GCE IP blocks from the new configuration

> Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process

I'm sure the devil is in the details, but generally speaking, these are 2 instances of critical code that gets exercised infrequently, which is a good place for bugs to hide.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#257

Post by DanielDent »

My post yesterday seems even more relevant today: https://news.ycombinator.com/item?id=11477552

It's a shame it's not easier or more common for people to create clones of (most|all) of their infrastructure for testing purposes.

Something like half of outages are caused by configuration oopsies.

If you accept that configuration is code, then you also come to the following disturbing conclusion: the usual test environment for critical network-related code in most environments is the production environment.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#259

Post by nothrabannosir »

seanp2k2 wrote: post #198
pjlegato wrote: post #123

Earlier quoted context omitted.

For this reason, most self-driving car development will only happen at large companies like Google, Tesla, Ford, etc. because they are the only ones who will be able to afford to purchase a massive general liability insurance policy.

I think that the other part with insurance is that insurers have no idea what the risk involved actually is (since it's not been around long) so they aim way high to cover themselves.

They should aim as high as they can; the price of a product is not bounded by its cost, but by what people are willing to pay.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#260

Post by pmarreck »

cjbprime wrote: post #17

It looks like there were at least three catastrophic bugs present: 1. Evaluated a configuration change before the change had finished syncing across all configuration files, resulting in rejecting the change. 2. So it tried to reject the change, but actually just deleted everything instead. 3. Something was supposed to catch changes that break everything, and it detected that everything was broken but its attempt to…

The take-home here is: Unit-test your failure states as well, people. Not just your happy paths!

I mean, this problem was a result of MULTIPLE untested failure states.

And yes, it IS possible to unit-test this sort of thing. You can fake out network connections and responses. I haven't yet found something that's impossible to unit-test, if you just think about how to do it properly, actually.

EDIT: Why downvotes without a typewritten rebuttal? That's just not what I expect from HN (as opposed to, say, Reddit)

Post reply on HN