Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

11–20 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#11

Post by Sanddancer »

wyldfire wrote: post #5

> . Internal monitors generated dozens of alerts in the seconds after the traffic loss became visible at 19:08 ... revert the most recent configuration changes ... the time from detection to decision to revert to the end of the outage was thus just 18 minutes. It's certainly good that they detected it as fast as they did. But I wonder if the fix time could be improved upon? Was the majority of that time spent discuss…

From the rest of the post, it sounds like replication time. Datacenters started dropping an hour beforehand one by one, and they had all fallen over by 19:08. Given that you have to push the rollback to routers around the world, and that peer routers have to propagate the changes from there, 18 minutes for a change like this sounds about right.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#12

Post by ddispaltro »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Agreed, I think a good postmortem distinguishes great companies from just good companies. The depth on philosophy, reasoning and then action is very digestible.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#13

Post by kyrra »

nzoschke wrote: post #4
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Seconded. Fast recovery of the problem, fast to publish a postmortem, and a very thurough postmortem. Outages suck, but are inevitable even for Google. With a response like this Google has gained even more trust from me.

Google's take on postmortems is really nice. As the SRE book points out, they are seen as a learning tool for others. Most internal postmortems are available for anyone within the company to see and learn from. As well, they are always blameless. No fingers are pointed at the person who caused the issue in the postmortem. They explain the issue, what happened, and how it can be prevented in the future.

Pair this with the outage tracking tools and you can find all the outages that have happened across Google and what caused them.

Then there is DiRT[0] testing to try and catch problems in a controlled manner. Having things break randomly through Google's infrastructure and you have to see if your service's setup and oncall people handle it properly is a really awesome exercise.

[0] http://queue.acm.org/detail.cfm?id=2371516

The opinions stated here are my own, not necessarily those of Google.

Edit: Changed from saying "all" to "most" postmortems being available to Googlers to see.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#14

Post by obulpathi »

> Finally, to underscore how seriously we are taking this event, we are offering GCE and VPN service credits to all impacted GCP applications equal to (respectively) 10% and 25% of their monthly charges for GCE and VPN.

These credits exceed what is promised by Google Cloud in their SLA's for Compute Engine and VPN service!

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#17

Post by cjbprime »

It looks like there were at least three catastrophic bugs present:

1. Evaluated a configuration change before the change had finished syncing across all configuration files, resulting in rejecting the change.

2. So it tried to reject the change, but actually just deleted everything instead.

3. Something was supposed to catch changes that break everything, and it detected that everything was broken but its attempt to do anything to fix it failed.

It is hard to imagine that this system has good test coverage.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#18

Post by fweespee_ch »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

> This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them?

Yes. However, the current failure rate of human drivers being improved on is the standard I care about.

http://www.cnbc.com/2015/10/29/crash-data-for-self-driving-c...

> After crunching the data, Schoettle and Sivak concluded there's an average of 9.1 crashes involving self-driving vehicles per million miles traveled. That's more than double the rate of 4.1 crashes per one million miles involving conventional vehicles.

That is the only number that matters to me. Google gets that to 4.0 per million miles and I'd say they are good to go.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#19

Post by dreamcompiler »

VonGuard wrote: post #6
reustle wrote: post #3

Earlier quoted context omitted.

I wouldn't worry so much. I'm sure self driving cars are going to save a lot more lives than they are going to end. Humans are terrible drivers, and the software will only get better.

Yeah, remember, auto-pilot in a plane needs to be 100% reliable, or everyone dies. A car needs to be, I dunno, 80%? Compared to a bad human driver, who still drives every damn day, a computer need only be about 60% reliable to be better. People suck at driving. Even a shitty self-driving car will save a ton of lives simply by obeying traffic laws.

An auto-pilot for an airplane is a considerably easier problem to solve. No lanes; no pedestrians; very little other traffic; three spatial degrees of freedom. That's why auto-pilots for airplanes have existed for almost a century but we're just now beginning to get self-driving cars. Humans are still better at dealing with the full panoply of crap that road driving throws at us.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#20

Post by jfoster »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

As long as the edge case bugs in self driving cars come up less frequently than human error, it's an overall improvement.
Post reply on HN