Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

301–310 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#301

Post by brlewis »

kevan wrote: post #169
drstewart wrote: post #30

Earlier quoted context omitted.

Seriously. This is a good postmortem, but these are hardly edge case bugs. In this case, major critical functionality just plain didn't work. Kind of shocking.

They explained the issues in laymans terms that most likely mask the true complexity of what happened. It's easy to read the final result: "tried to reject but then deleted everything" and think "Well duh that's bad, who would build a system that does that?", but I think you're fooling yourself if you think that edge cases couldn't cause that.

Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.

It seems obvious to me that the push system should not proceed without confirmation from the management software, and the management software should not confirm the change is OK if it detects failure.

I see a straightforward defect here, not a confluence of edge cases.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#302

Post by mmel »

contingencies wrote: post #247

TLDR; they simply didn't test their (global!) custom route announcement management software. An edge case was triggered in production, and they gee-whiz-automatically went offline. Epic fail. PS. To the downvoters, truth hurts.

I think you're getting downvoted due to the snarky tone more than any "truth" you are stating.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#303

Post by nostalgiac »

fizzbatter wrote: post #73
llamataboot wrote: post #58

Earlier quoted context omitted.

My consolation in that fact is that weird edge cases happen with human driven cars as well. Someone has a seizure and crashes, or more commonly reaches for a cigarette, the radio, their phone. People hit ice or water and overcorrect their spin. People drive too fast. Etc etc etc. Not even all edge cases, many common modes of failure. I except self-driving cars that kill people will be a huge emotional issue for a lot…

Yup. People seem to be overly critical with automated car failures. Personally, i think automated cars are going to easily be better than humans in the working cases (both human and ai are concious) . Next, i expect to see fully operational backup systems. Eg, if a monitoring system decides that the primary system is failing for whatever reason, be it bug or unhandled road condition (tree/etc), the backup system take…

Sounds good in theory. Until the bug that causes failure is also present in the monitoring system, and as such doesn't fail over to the backup system. AKA exactly what happened here to Google.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#304

Post by vacri »

dreamcompiler wrote: post #278
vacri wrote: post #232

Earlier quoted context omitted.

Autopilot in a plane usually doesn't involve autonavigation, whereas autodrive in a car generally requires navigation. Autodrive in a car without navigation is basically 'cruise control'.

'Cruise control' does not (at least it didn't until very recently) even attempt to avoid collisions with neighboring cars or keep the car in its lane. Even without navigation, autodrive in a car is a considerably more difficult problem than either cruise control or autopilot in a plane.

My point is that 'autopilot' is more like 'cruise control', and except in advanced cases, is not analagous to 'autodrive'.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#305

Post by contingencies »

mmel wrote: post #302
contingencies wrote: post #247

TLDR; they simply didn't test their (global!) custom route announcement management software. An edge case was triggered in production, and they gee-whiz-automatically went offline. Epic fail. PS. To the downvoters, truth hurts.

I think you're getting downvoted due to the snarky tone more than any "truth" you are stating.

Well, how to phrase the same thing briefly without sounding snarky?

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#306

Post by campers »

nostrademons wrote: post #113
teraflop wrote: post #31

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ... This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007: > In closing we point out a challenge that we faced in testing our system for which we have no systemat…

Google actually does have a systematic solution: fault injection. Google's systems are designed so that you can (manually, if you have the right privileges) tell an RPC to fail regardless of whether it would otherwise have succeeded, and then test the response of the system as a whole. The problem is that these failure cases are exercised much less frequently than the "normal execution" code paths are. For example, e…

There was an relevant section in the Google SRE book notes posted here the other day about injecting faults into Chubby, their distributed lock service, which was too reliable.

Ex: Chubby planned outages Google found that Chubby was consistently over its SLO, and that global Chubby outages would cause unusually bad outages at Google Chubby was so reliable that teams were incorrectly assuming that it would never be down and failing to design systems that account for failures in Chubby Solution: take Chubby down globally when it’s too far above its SLO for a quarter to “show” teams that Chubby can go down

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#307

Post by qaq »

creshal wrote: post #213
seanp2k2 wrote: post #179

Earlier quoted context omitted.

Car manufacturers are some of the last people I trust to be doing software updates. The recent Takata airbag recall is an example of the ensuing fecal tornado from large recalls: http://blog.caranddriver.com/massive-takata-airbag-recall-ev... In some cases, people are having to wait months to get new airbags because they just don't have them in stock. In the computer case, would you want to keep driving until they ca…

A friend of mine works for VW's engine computer division. Yes, those engine computers. After all I've heard of their development methods (or lack thereof), I'm surprised the engines even start more often than one time out of ten.

Care to elaborate? Kernel developers are not practising SCRUM or TDD yet are shipping a fairly stable product.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#308

Post by distrill »

gliderShip wrote: post #107
ben_jones wrote: post #64

Earlier quoted context omitted.

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

I think it still is that simple. Clearly killing millions is not better than human control, so would be unacceptable.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#310

Post by lugg »

packetslave wrote: post #294
lugg wrote: post #282

Earlier quoted context omitted.

This wasn't an edge case. It was two bugs in two sections of code both designed to recover from a serious problem. It sounds like both sections of code were not tested properly at the very least. Sounds to me like someone just didn't bother to test the failsafe part of the code.

...and you're basing this on what, exactly? It's easy to pontificate about what people "didn't bother to test" based on zero information.

First bug:

In a failure case, it should remove the failing config, not all of them.

Pretty hard thing to miss if you test for it with any level of basic unit test or similar.

Second bug: canary failure should prevent further propogation of the bad config.

A little more difficult to test with automated tests due to requiring a connection. It sounds like this was in fact tested, but the usage between the two bits of software was not tested. A good integration test would have caught this. But I wouldn't call that required. I would at least however think it was required that the use case of that particular code to be at least manually checked because, you know it's a feature for disaster prevention / recovery.

There was enough information to deduce this pretty easily. Although they did tend to glaze over it in the write-up, almost purposefully.

For all those spouting that this was a good postmortem, not really, it's a good covering of ones ass, a good spin, sidestepping the real root cause.

What has slas and "here take credits" got to do with a postmortem?

I'm not really sure why I got downvoted for this. The post mortem was good but it wasn't something I'd aim to strive for. I like gcloud and I'll keep using it but I find the response to this thing a little bit hard to swallow.

Post reply on HN