Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

61–70 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#61

Post by daveguy »

dgacmu wrote: post #54
daveguy wrote: post #42

Earlier quoted context omitted.

Based on the higher level status page: https://status.cloud.google.com/summary It looks like GCE uptime is well below four 9's reliability for a sliding 1 year timeframe.

Traynor was quoted in a networkworld article last year saying they aim for three and a half nines (99.95%). But you need to read into the incidents more carefully -- figuring out actual "uptime" is quite hard. Consider the longest-lasting incident: "On Tuesday 23 February 2016, for a duration of 10 hours and 6 minutes, 7.8% of Google Compute Engine projects had reduced quotas. ... Any resources that were already crea…

Agreed. It is difficult to tell. But if the bug is preventing you from processing (because you can't save the existing results) then it's essentially down time for new processing. There are also connectivity issues by region and DNS issues. It is difficult to get exact downtime considering partial failures.

That said, this is the second major asia-east1 downtime in 90 days:

https://status.cloud.google.com/incident/compute/16002

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#62

Post by maxander »

ikeboy wrote: post #44
mjibson wrote: post #27

Earlier quoted context omitted.

I'm attempting to even imagine how one would build a useful way to test this. Would they have to have a secondary, world-wide datacenter network with all their various services behind it?

You could have it send messages to the actual servers, but with an added flag that says "fake", which makes the servers ignore the message/send back a message saying pass/fail/whatever (testing the flag could happen first, one server at a time manually). Then check whether the program continued to push updates.

You may be able to build an elaborate system of dummy network operations to test with, but this system may wind up with bugs that mask what would be errors in the real system. And how to you test against that? A dummy network to test the dummy network operations on? What if the dummy network contains bugs that make it behave significantly different from the real network, in error cases? How do you test for that?

Its turtles all the way down!

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#63

Post by msellout »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Humans also make mistakes in corner cases. I'm no more afraid of an auto-steering car than a human-steered car.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#64

Post by ben_jones »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1].

The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that might actually happen to me!

[1]: https://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_i...

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#65

Post by stcredzero »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them?

Formal systems?

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#66

Post by eloff »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Yes but how many people drive stoned, drunk, or distracted?

How many people drive aggressively, speeding, or erratically? How many people do dumb things on the road?

As a software engineer I know that there will be bugs and some will likely kill people. But as a driver who has driven many years in less civilized countries, I know that human beings are terrible drivers.

Who would you rather share the road with, computer drivers that drive like your grandma, or a bunch of humans? It's a no-brainer right?

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#67

Post by duaneb »

VonGuard wrote: post #6
reustle wrote: post #3

Earlier quoted context omitted.

I wouldn't worry so much. I'm sure self driving cars are going to save a lot more lives than they are going to end. Humans are terrible drivers, and the software will only get better.

Yeah, remember, auto-pilot in a plane needs to be 100% reliable, or everyone dies. A car needs to be, I dunno, 80%? Compared to a bad human driver, who still drives every damn day, a computer need only be about 60% reliable to be better. People suck at driving. Even a shitty self-driving car will save a ton of lives simply by obeying traffic laws.

But autopilot for planes is actually much easier than negotiating traffic with irrational humans with road rage. You can coordinate with air traffic for takeoffs and landings, and there is very little to run into at tens of thousands of feet.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#68

Post by llamataboot »

boydc wrote: post #22
VonGuard wrote: post #6

Earlier quoted context omitted.

Yeah, remember, auto-pilot in a plane needs to be 100% reliable, or everyone dies. A car needs to be, I dunno, 80%? Compared to a bad human driver, who still drives every damn day, a computer need only be about 60% reliable to be better. People suck at driving. Even a shitty self-driving car will save a ton of lives simply by obeying traffic laws.

Self-driving car could be better than human in average. But as long as there are human drivers who drive better than self-driving software, it would be disaster for these drivers. We definitely do not want some technique than do good for majority but do horrible things for minority, right?

This would be true only if your driving ability only affected your chance to die, but your driving ability has an effect on everyone else's safety on the road as well!

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#69

Post by llamataboot »

ben_jones wrote: post #64
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

No because negligent driving doesn't just put the driver at risk - it puts everyone else on the road plus pedestrians at risk

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#70

Post by duaneb »

infinotize wrote: post #50
dreamcompiler wrote: post #19

Earlier quoted context omitted.

An auto-pilot for an airplane is a considerably easier problem to solve. No lanes; no pedestrians; very little other traffic; three spatial degrees of freedom. That's why auto-pilots for airplanes have existed for almost a century but we're just now beginning to get self-driving cars. Humans are still better at dealing with the full panoply of crap that road driving throws at us.

Aircraft autopilots also rely on experienced and licensed pilots to operate them and be responsible for the aircraft at all times. Self driving cars have assume the operator is not particularly capable nor paying attention to anything happening on the road.

Do they rely on the pilot? I was under the impression it was entirely hands off.
Post reply on HN