Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

211–220 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#211

Post by heisnotanalien »

erichocean wrote: post #163
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Are BGP updates for Google's own router configurations really so frequent that they can't pay an engineer to at least monitor the propagation of configuration changes? In this case, a human would have instantly seen that the update was a) rejected (as explained in the postmortem), and b) holy shit, WHY DID THE ROUTER CHANGE ITS OWN CONFIGURATION TO BLOW AWAY ALL OF THE GCE ROUTES!?! I'm all for automation, but WTF? I…

But a computer could do the same thing. It would be possible to alert on all the routes being blown away?

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#212

Post by songgao »

mikecb wrote: post #112
icebraining wrote: post #35

Earlier quoted context omitted.

Only barely. They're down to 2.5 minutes of downtime left for the next 30 days if they want to keep the 99.95% level.

They do SLO by quarter.

Just to clarify, the criteria is using "monthly uptime percentage": https://cloud.google.com/compute/sla

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#213

Post by creshal »

seanp2k2 wrote: post #179
sriram_sun wrote: post #141

Earlier quoted context omitted.

Car manufacturers conduct recalls all the time. There might be the possibility that a million self-driving cars will be held hostage from a remote control tower simultaneously leading to injury or death to millions. However, in practice, as soon as an issue is discovered, there will be the equivalent of recalls (remote updates) and things like this will be fixed. People who are uncomfortable with self driving cars wi…

Car manufacturers are some of the last people I trust to be doing software updates. The recent Takata airbag recall is an example of the ensuing fecal tornado from large recalls: http://blog.caranddriver.com/massive-takata-airbag-recall-ev... In some cases, people are having to wait months to get new airbags because they just don't have them in stock. In the computer case, would you want to keep driving until they ca…

A friend of mine works for VW's engine computer division. Yes, those engine computers. After all I've heard of their development methods (or lack thereof), I'm surprised the engines even start more often than one time out of ten.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#214

Post by nxzero »

Gravityloss wrote: post #205
nxzero wrote: post #192

Earlier quoted context omitted.

Comparing the risk of a live update to a system lives depend on to the risk of some Google services going down is irrational. At some point, delaying the deployment of updates system wide would cause more, not less risks.

There are businesses that fit somewhere between Boeing and Spotify where failures still have some kind of steeper than casual cost. On Hacker News the "move fast and break things" ethos is probably making sense for many of the people submitting and commenting, since their business is closer to casual usage anyway. But that's not the whole audience.

Shit happens, when it comes to engineering, I'd trust Google more than even likely Boeing to manage systemic risk.

As for cars, it's a real risk, but not the same as the bugs Google experienced; I personally have experienced a "bug" driving a car at high speeds, which resulted in a number of major electronic systems failing due to custom systems installed by a well known US startup.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#215

Post by essayist »

essayist wrote: post #148
gliderShip wrote: post #107

Earlier quoted context omitted.

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

Fun. Johnny555 is right of course, and it was the Jeep hacked on the Interstate last year that first inspired me. The airplane scenario may be more likely, but I liked the additive nature of so many cars, cars a bit like insects. (You could go further in this direction by conjuring up multiple drone swarms.)

As to movie points: of course Will Smith is the hero, and we'll handle DC rush hour stasis through special effects. ;-)

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#216

Post by rdtsc »

> However, in this instance a previously-unseen software bug was triggered, and instead of retaining the previous known good configuration, the management software instead removed all GCE IP blocks from the new configuration and began to push this new, incomplete configuration to the network.

Always test your crash / exception handling / special case termination+recovery code in production.

I have seen this too often. Most often in in "every day" cases when service has a "nice" catch way of stopping and recovering. Then has a separate "if killed by SIGKILL/immediate power failure" crash and recovery. This last bit never gets tests and run in production.

One day power failure happens, service restart and tries to recover. Code that almost never runs, now runs and the whole thing goes into an unknown broken state.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#217

Post by dpark »

seanp2k2 wrote: post #179
sriram_sun wrote: post #141

Earlier quoted context omitted.

Car manufacturers conduct recalls all the time. There might be the possibility that a million self-driving cars will be held hostage from a remote control tower simultaneously leading to injury or death to millions. However, in practice, as soon as an issue is discovered, there will be the equivalent of recalls (remote updates) and things like this will be fixed. People who are uncomfortable with self driving cars wi…

Car manufacturers are some of the last people I trust to be doing software updates. The recent Takata airbag recall is an example of the ensuing fecal tornado from large recalls: http://blog.caranddriver.com/massive-takata-airbag-recall-ev... In some cases, people are having to wait months to get new airbags because they just don't have them in stock. In the computer case, would you want to keep driving until they ca…

> Car manufacturers are some of the last people I trust to be doing software updates. The recent Takata airbag recall is an example of the ensuing fecal tornado from large recalls: http://blog.caranddriver.com/massive-takata-airbag-recall-ev...

>In some cases, people are having to wait months to get new airbags because they just don't have them in stock. In the computer case, would you want to keep driving until they can get you scheduled for a software update? Remember that many cars can't update critical software OTA.*

So I assume you don't own a car and you avoid them at all costs? Otherwise your paranoia becomes hypocrisy. If you cannot trust the car company to deliver software updates, you can't trust them to write the software in the first place, and modern cars are full of safety-critical software.

I also don't know why you're equating the a manufacturing capacity limitation with a software update limitation. It's not as if Toyota is going to have trouble shipping bits a million times vs a thousand times once the software update is written.

I think we can also safely assume that self-driving cars will generally be updatable OTA. But yes, you could drive it to the dealer if needed, and worst case the dealer could send people on-site to do the update.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#219

Post by 794CD01 »

ThrustVectoring wrote: post #88
794CD01 wrote: post #80

Earlier quoted context omitted.

They don't have to be safer than driving is today. They can be significantly less safe while still being an improvement for society because drivers will be able to focus on other activities while travelling instead of wasting that time focusing on driving the car.

There's psychological and game theoretic factors that the safety has to overcome in order to be acceptable. Part of why human drivers are allowed today is because the people who bear the cost of driving decisions are directly involved in making those decisions. Once you give up control to a third party, they need to be significantly better to make it an acceptable choice on the individual level. In other words, I agr…

People in general don't willingly prioritize their own safety at the expense of convenience. That's why cars beep when you don't buckle your seatbelt. That's why people drink soda and eat fast food.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#220

Post by trhway »

as devops Borat was saying all along, automated propagation of a error as the main root cause here. A error (new configuration) should be rolled out site by site - ok us-east1, move onto us-west1 ... ok, move onto ... . A canary site may be the first in sequence, yet success ("no failure reported") can't be a big "ok" for automated push to all sites at the same time.
Post reply on HN