Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

41–50 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#41

Post by jchrisa »

boydc wrote: post #22
VonGuard wrote: post #6

Earlier quoted context omitted.

Yeah, remember, auto-pilot in a plane needs to be 100% reliable, or everyone dies. A car needs to be, I dunno, 80%? Compared to a bad human driver, who still drives every damn day, a computer need only be about 60% reliable to be better. People suck at driving. Even a shitty self-driving car will save a ton of lives simply by obeying traffic laws.

Self-driving car could be better than human in average. But as long as there are human drivers who drive better than self-driving software, it would be disaster for these drivers. We definitely do not want some technique than do good for majority but do horrible things for minority, right?

Why would it be a disaster?

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#42

Post by daveguy »

balls187 wrote: post #10

Nice post mortem. That outtage gives GCE at best a four 9's reliability for 2016.

Based on the higher level status page:

https://status.cloud.google.com/summary

It looks like GCE uptime is well below four 9's reliability for a sliding 1 year timeframe.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#43

Post by nxzero »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Idea that edge cases in autonomous vehicles would result in 30,000+ deaths a year to me is a stretch.

If you dispute this, please explain.

If your position is that one death is too many, that is illogical relative to the option of letting people drive cars.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#44

Post by ikeboy »

mjibson wrote: post #27
cjbprime wrote: post #17

It looks like there were at least three catastrophic bugs present: 1. Evaluated a configuration change before the change had finished syncing across all configuration files, resulting in rejecting the change. 2. So it tried to reject the change, but actually just deleted everything instead. 3. Something was supposed to catch changes that break everything, and it detected that everything was broken but its attempt to…

I'm attempting to even imagine how one would build a useful way to test this. Would they have to have a secondary, world-wide datacenter network with all their various services behind it?

You could have it send messages to the actual servers, but with an added flag that says "fake", which makes the servers ignore the message/send back a message saying pass/fail/whatever (testing the flag could happen first, one server at a time manually). Then check whether the program continued to push updates.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#46

Post by ksou32 »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Humans have bugs all the time, you can just faint for no reason while driving . ..

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#49

Post by kohanz »

brianwawok wrote: post #33
reustle wrote: post #3

Earlier quoted context omitted.

I wouldn't worry so much. I'm sure self driving cars are going to save a lot more lives than they are going to end. Humans are terrible drivers, and the software will only get better.

I know this logically. But emotionally I know how many bugs I have written in my life. I know software devs are human.. aka I know how the sausage is made.

Yes, but have you been part of the development & testing effort for mission-critical software (e.g. a class 1 or 2 medical device?). It's not true in all cases, but for the most part the level of QA that goes into the devices before release is significantly higher than that of your average product. This is why regulation is required.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#50

Post by infinotize »

dreamcompiler wrote: post #19
VonGuard wrote: post #6

Earlier quoted context omitted.

Yeah, remember, auto-pilot in a plane needs to be 100% reliable, or everyone dies. A car needs to be, I dunno, 80%? Compared to a bad human driver, who still drives every damn day, a computer need only be about 60% reliable to be better. People suck at driving. Even a shitty self-driving car will save a ton of lives simply by obeying traffic laws.

An auto-pilot for an airplane is a considerably easier problem to solve. No lanes; no pedestrians; very little other traffic; three spatial degrees of freedom. That's why auto-pilots for airplanes have existed for almost a century but we're just now beginning to get self-driving cars. Humans are still better at dealing with the full panoply of crap that road driving throws at us.

Aircraft autopilots also rely on experienced and licensed pilots to operate them and be responsible for the aircraft at all times. Self driving cars have assume the operator is not particularly capable nor paying attention to anything happening on the road.
Post reply on HN