Post-Mortem for Google Compute Engine’s Global Outage on April 11
status.cloud.google.com
Post-Mortem for Google Compute Engine’s Global Outage on April 11
1–10 of 368 posts
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#2This is a very good Post-Mortem.
As I assumed it was kind of a corner case bug meet corner case bug met corner case bug.
This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them?
Making software is hard....
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#3This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....
I wouldn't worry so much. I'm sure self driving cars are going to save a lot more lives than they are going to end. Humans are terrible drivers, and the software will only get better.
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#4This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....
Seconded. Fast recovery of the problem, fast to publish a postmortem, and a very thurough postmortem.
Outages suck, but are inevitable even for Google. With a response like this Google has gained even more trust from me.
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#5> . Internal monitors generated dozens of alerts in the seconds after the traffic loss became visible at 19:08 ... revert the most recent configuration changes ... the time from detection to decision to revert to the end of the outage was thus just 18 minutes.
It's certainly good that they detected it as fast as they did. But I wonder if the fix time could be improved upon? Was the majority of that time spent discussing the corrective action to be taken? Or does it take that much time to replicate the fix?
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#6This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....
I wouldn't worry so much. I'm sure self driving cars are going to save a lot more lives than they are going to end. Humans are terrible drivers, and the software will only get better.
Yeah, remember, auto-pilot in a plane needs to be 100% reliable, or everyone dies. A car needs to be, I dunno, 80%? Compared to a bad human driver, who still drives every damn day, a computer need only be about 60% reliable to be better.
People suck at driving. Even a shitty self-driving car will save a ton of lives simply by obeying traffic laws.
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#7This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....
self driving cars will be safer than BGP that' s for sure.
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#8>However, in this instance a previously-unseen software bug was triggered, and instead of retaining the previous known good configuration, the management software instead removed all GCE IP blocks from the new configuration and began to push this new, incomplete configuration to the network.
>Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.
I assume the software was originally tested to make sure it works in case of failure. It would be interesting to know exactly what the bug was and why it didn't show in tests.
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#9> . Internal monitors generated dozens of alerts in the seconds after the traffic loss became visible at 19:08 ... revert the most recent configuration changes ... the time from detection to decision to revert to the end of the outage was thus just 18 minutes. It's certainly good that they detected it as fast as they did. But I wonder if the fix time could be improved upon? Was the majority of that time spent discuss…
> But I wonder if the fix time could be improved upon?
Rushing to enact a solution can sometimes exacerbate the problem.
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#10Nice post mortem.
That outtage gives GCE at best a four 9's reliability for 2016.