Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

201–210 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#201

Post by pyre »

ben_jones wrote: post #64
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

Well, this bug took down the entire system. What happens when self-driving software hits a similar bug? I don't think that there is any precedent for that sort of thing with manually driven cars. The scale could easily be larger than 100-car pile-ups due to poor weather conditions.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#202

Post by Johnny555 »

essayist wrote: post #148
gliderShip wrote: post #107

Earlier quoted context omitted.

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

You don't need self driving cars for such a scenario to happen -- cars are increasingly drive by wire, and driver assistance features being added to cars (automatic lane keeping, automatic breaking, smart cruise control, etc) mean computers are already capable of taking over cars.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#203

Post by kohanz »

flurdy wrote: post #178
praxulus wrote: post #83

Earlier quoted context omitted.

GCE downtime just means people lose money, it's not life-or-death. Skimping on QA in order to reduce costs and get to market faster is a perfectly reasonable decision when the consequences are so mundane.

I understand what you mean but that is generalising too much what people use GCE, public clouds, self hosted servers for, and especially going forward. It is not all convenience applications, game backends etc. What people these days use AWS/GCE for is so varied, even public sector use AWS Gov Region for example. Downtime consequences is not just money lost but can be life-and-death and for some application they need…

You make a good point, but in the end the responsibility is on the life-critical application (e.g. medical software, device, self-driving car) to ensure that is has been properly QA'd and that all of its dependencies (including any cloud services or framework that it is built upon) meet its safety requirements. The event of an app server or cloud service experiencing downtime would very much have to be planned for as part of a Risk Management exercise. Ignoring that possibility would be negligent.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#204

Post by seanp2k2 »

amelius wrote: post #104
ThrustVectoring wrote: post #88

Earlier quoted context omitted.

There's psychological and game theoretic factors that the safety has to overcome in order to be acceptable. Part of why human drivers are allowed today is because the people who bear the cost of driving decisions are directly involved in making those decisions. Once you give up control to a third party, they need to be significantly better to make it an acceptable choice on the individual level. In other words, I agr…

> Part of why human drivers are allowed today is because the people who bear the cost of driving decisions are directly involved in making those decisions. This gives me weird visions of Google engineers with a necklace that explodes in the event that one of their cars causes an accident :S

The difference there is that one person's mistake driving causes one accident. One person's mistake programming might cause a thousand.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#205

Post by Gravityloss »

nxzero wrote: post #192
Gravityloss wrote: post #181

I'm waiting for the time when they push over the air updates to airplanes in flight. "You can fly safely, we have canaries and staged deployment" A year forward: "Unfortunately because the canary verification as well as the staged deployment code was broken, instead of one crash and 300 dead, an update was pushed to all aircraft, which subsequently caused them to crash, killing 70,000 people." I'm not 100% sure why t…

Comparing the risk of a live update to a system lives depend on to the risk of some Google services going down is irrational. At some point, delaying the deployment of updates system wide would cause more, not less risks.

There are businesses that fit somewhere between Boeing and Spotify where failures still have some kind of steeper than casual cost.

On Hacker News the "move fast and break things" ethos is probably making sense for many of the people submitting and commenting, since their business is closer to casual usage anyway. But that's not the whole audience.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#206

Post by pavel_lishin »

blackflame7000 wrote: post #191
pavel_lishin wrote: post #176

Earlier quoted context omitted.

> hacking into commercial airliners auto-pilot systems, and directing dozens of flights onto a target. I would imagine that any pilot would figure out what was going on, unless it was on an incredibly foggy day.

It's Hollywood, name one movie where the villain did not disable the manual override. That's villainy 101.

Sure, but rayval was talking about a scenario that could happen today.

Although looking at the other comments, I think I'm significantly underestimating just how much of modern airliners is dependent on software. The pilots might be able to see that they're heading for disaster, but may not be able to do anything about it.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#207

Post by lfowles »

brianwawok wrote: post #93
bcook wrote: post #52

Earlier quoted context omitted.

Why are you comparing self-driving cars to exclusively a "paying attention driver"? For self-driving cars to be safer than human drivers, there is no requirement that the self-driving cars should be better/safer than the best human driver... the self-driving car simply needs to be safer than the majority of humans.

> For self-driving cars to be safer than human drivers, there is no requirement that the self-driving cars should be better/safer than the best human driver... the self-driving car simply needs to be safer than the majority of humans. That is true on a whole, but not true for ME. It needs to be safer than ME, not some hypotehtical average person. Further compounding it: > For driving skills, 93% of the U.S. sample an…

Consider that most driving is done feet away from another vehicle. If those cars start being replaced by self-driving cars, then the you are safer (system safety x personal safety). The car might make more dubious decisions than you would yourself, but now it has less opportunities for failure due to others driving erratically.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#208

Post by dagss »

brianwawok wrote: post #93
bcook wrote: post #52

Earlier quoted context omitted.

Why are you comparing self-driving cars to exclusively a "paying attention driver"? For self-driving cars to be safer than human drivers, there is no requirement that the self-driving cars should be better/safer than the best human driver... the self-driving car simply needs to be safer than the majority of humans.

> For self-driving cars to be safer than human drivers, there is no requirement that the self-driving cars should be better/safer than the best human driver... the self-driving car simply needs to be safer than the majority of humans. That is true on a whole, but not true for ME. It needs to be safer than ME, not some hypotehtical average person. Further compounding it: > For driving skills, 93% of the U.S. sample an…

Somebody can crash into you.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#209

Post by fixermark »

stcredzero wrote: post #98

In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout. Classic Two Generals. "No news is good news," generally isn't a good design philo…

It may not be good design, but it might be necessary / practical design. If you have enough machines that some percentage of them are down or unreachable at any given time, you can't wait for full go-ahead before proceeding; you'll never get full go-ahead. So you're left with probabilistic solutions, and as T approaches infinity the expectation of more than zero false-positives approaches 1.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#210

Post by fixermark »

qaq wrote: post #24

DRY "The inconsistency was triggered by a timing quirk in the IP block removal - the IP block had been removed from one configuration file, but this change had not yet propagated to a second configuration file also used in network configuration management."

DRY is tougher when for practical reasons data must be physically cached locally.
Post reply on HN