Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

341–350 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#341

Post by tremon »

stcredzero wrote: post #65
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Formal systems?

Formal systems are still built on a model of the outside world, not on the world itself FAFAIK. Even if your formal coverage is 100%, you can not anticipate all weird edge cases the real world can come up with.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#342

Post by dfox »

sk5t wrote: post #288
seanp2k2 wrote: post #175

Earlier quoted context omitted.

In the airplane case, it's possible today: https://m.youtube.com/watch?v=CXv1j3GbgLk And https://m.youtube.com/watch?v=Uy3nXXZgqmg TL;DR you simulate a bunch of other planes in close proximity and the auto-pilot freaks out and tries to avoid them. As the second talk explains, the pilots would definitely notice and switch autopilot off. This is why IMO it's very important to not take ultimate control away from humans…

A minor point, but the electronic accelerator control in autos is called "throttle-by-wire."

I've always seen that called EPC for Electronic Pedal Control, but that is probably VW-ism.

On the other hand on EFI car, having mechanical throttle cable does not add much to hack-safety as the ECU always has some way to override closed throttle (either disengaging throttle pedal mechanically switches the control of throttle to ECU operated servo or there is completely separate throttle controlled by ECU).

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#343

Post by dfox »

lallysingh wrote: post #117
deegles wrote: post #106

Earlier quoted context omitted.

This would have a dramatic chilling effect on hiring for self-driving car software developers, which would ironically make them less safe.

It might lead to some decent work on formal verification of programs.

The problem with formal verification of systems like this is that checking whether software matches the specification or whether the specification is self-consistent is not the interesting problem. Whether specification matches the real world is the interesting problem and there is no formal way to verify that.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#344

Post by dfox »

brandmeyer wrote: post #138
teraflop wrote: post #31

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ... This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007: > In closing we point out a challenge that we faced in testing our system for which we have no systemat…

The standard solution in realtime safety-critical systems is to perform health monitoring in addition to robust fallbacks, such that when the system is falling back, it is reported as unhealthy. For example, the CAN bus normally has an automatic retry feature on a variety of errors. A properly functioning CAN bus should have a bit error rate that is nearly zero. Lightly loaded, it can tolerate a very high error rate…

CAN and it's automatic retransmit is actually pretty good example of how simple transient problems can quickly overgrow into global system failures. On typical CAN the bandwidth headroom is small enough that when all colliding/failed telegrams would be blindly retransmitted the collision rate would skyrocket and thus only high-priority traffic will make any progress, and as on CAN priority and purpose is intrinsically linked together, from the global point of view nothing will make progress. That's why most CAN controllers have configurable retransmit behavior per packet (drop/retry/raise error and application deal with that) and partially why today's cars have multiple CAN buses.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#345

Post by infinite8s »

EmployedRussian wrote: post #323
seanp2k2 wrote: post #175

Earlier quoted context omitted.

In the airplane case, it's possible today: https://m.youtube.com/watch?v=CXv1j3GbgLk And https://m.youtube.com/watch?v=Uy3nXXZgqmg TL;DR you simulate a bunch of other planes in close proximity and the auto-pilot freaks out and tries to avoid them. As the second talk explains, the pilots would definitely notice and switch autopilot off. This is why IMO it's very important to not take ultimate control away from humans…

> I would personally never buy one of the Google (or any other) self-driving models with no controls. Google cars have the Big Red Button, which shuts off self-driving system and brings the car to a stop. What more controls do you need?

When are you barreling down a highway at 65 miles per hour, turning off the car might not be the best solution.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#346

Post by ceejayoz »

neurotech1 wrote: post #317
ceejayoz wrote: post #127

Earlier quoted context omitted.

All pilots are trained to recover from a stall. A failed autopilot could look like all sorts of things, from just automatically disconnecting itself (usually with a loud warning alert) to issuing incorrect instructions (which is why the pilots are supposed to be awake and alert while it's engaged, watching the instruments).

Actually a key finding in AF447 was that pilots were not trained on how to recognize and recover from a high altitude stall. It is not like flying a Cessna 150. The junior first officer didn't realize the aircraft had stalled. Pilots were trained on the procedure for recovering from a low altitude stall; 100% or TOGA thrust and power out of it while minimizing altitude loss. Training has now changed for both low alti…

IIRC, part of the issue was that the two pilots issued contradictory joystick commands, which the plane averaged to zero. Which is a bit terrifying.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#348

Post by hinkley »

gliderShip wrote: post #107
ben_jones wrote: post #64

Earlier quoted context omitted.

Self driving cars don't have to be perfect. They just have to be safer then driving is today [1]. The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that mi…

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

Imagine a leap second bug causing vehicles all over the US to crash into NYE revellers.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#349

Post by dylanz »

Completely off topic, but this thread is an example of why I (and a lot of people) want collapsible comments native to HN. I'm on my phone, in Safari, and I had to scroll for over 20 seconds just to reach the second comment. The first comment was a tangent about self-driving cars, which while relevant, I didn't want to read about.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#350

Post by OldSchoolJohnny »

dylanz wrote: post #349

Completely off topic, but this thread is an example of why I (and a lot of people) want collapsible comments native to HN. I'm on my phone, in Safari, and I had to scroll for over 20 seconds just to reach the second comment. The first comment was a tangent about self-driving cars, which while relevant, I didn't want to read about.

Especially considering that nearly every post on HN features an often tangential first comment that goes on and on and on...
Post reply on HN