Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

291–300 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#291

Post by thisisnotatest »

hashkb wrote: post #233
Matt3o12_ wrote: post #231

Earlier quoted context omitted.

There is something that's called fail safe(ly). In case of any error inside the car, it should sowly decelerate and pull over. Sure, it will cause a lot of traffic problems if say 10% of all cars did that at the same time but the damage would not be as severe as a ghost driver entering the freeway with 140mph. There are a few instances where some bad tesla batteries (the standard 12 volt batteries ironically) failed,…

Google had fail safes which failed. What if a million cars pull over to the left instead of the right as a failed failsafe? Willing to risk your life?

Yes, I'm eager to risk my life on self-driving car failures; it would be a tremendous step up from risking my life on human drivers (including myself!) as I do on a daily basis.

I'm also a biker. In 2013, 4,735 pedestrians and 743 bicyclists were killed in crashes with motor vehicles. http://www.pedbikeinfo.org/data/factsheet_crash.cfm

In the future when self-driving or at least augmented driving is commonplace, I hope that number will be a lot lower.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#292

Post by [deleted] »

mjibson wrote: post #27
cjbprime wrote: post #17

It looks like there were at least three catastrophic bugs present: 1. Evaluated a configuration change before the change had finished syncing across all configuration files, resulting in rejecting the change. 2. So it tried to reject the change, but actually just deleted everything instead. 3. Something was supposed to catch changes that break everything, and it detected that everything was broken but its attempt to…

I'm attempting to even imagine how one would build a useful way to test this. Would they have to have a secondary, world-wide datacenter network with all their various services behind it?

[deleted]

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#293

Post by DanielDent »

aiiane wrote: post #268
DanielDent wrote: post #257

My post yesterday seems even more relevant today: https://news.ycombinator.com/item?id=11477552 It's a shame it's not easier or more common for people to create clones of (most|all) of their infrastructure for testing purposes. Something like half of outages are caused by configuration oopsies. If you accept that configuration is code, then you also come to the following disturbing conclusion: the usual test environm…

The main issue there is that "environments" are defined by configuration, so if you try to set up a configuration test environment, you run into a direct logical impass: either your configs are production configs, and thus not a separate environment, or they're different from production configs, and thus may provide different test results from production.

While I agree with you, I think we could get closer to "production" than is common right now.

In an AWS environment, imagine a setup where all that differs is the API keys used (the API keys of the production vs test environment). What gets tricky is dealing with external dependencies, user data, and simulating traffic.

For an example more relevant to today's issue: imagine a second simulated "internet" in a globally distributed lab environment. With BGP configs, fake external BGP sessions, etc, servers receiving production traffic, etc.

I get that it's a lot of work to setup and would require ongoing work to maintain - and that it's hard/impossible to have it correctly simulate the many nuances of real world traffic - and yet I also think in many cases it would be sufficient to prevent issues from making it into production.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#294

Post by packetslave »

lugg wrote: post #282
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

This wasn't an edge case. It was two bugs in two sections of code both designed to recover from a serious problem. It sounds like both sections of code were not tested properly at the very least. Sounds to me like someone just didn't bother to test the failsafe part of the code.

...and you're basing this on what, exactly? It's easy to pontificate about what people "didn't bother to test" based on zero information.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#295

Post by Swannie »

poooogles wrote: post #53
eranation wrote: post #32

This is very interesting. From the little I understand (sorry for using AWS terms as I am more versed with AWS than GCE) this can happen to AWS as well right? even if your software is deployed to multiple AZs / multiple regions, if bad routing / network configuration makes it through the various protection mechanisms then basically no amount of redundancy can help if your service is part of the non functional IP bloc…

I'm not sure the AWS network follows the same setup, AWS has very distinct blocks between the US/EU/APAC compared to GCP where you can inherit the same IP if you quickly delete/recreate instances in different regions?

I was going to post the same comment too.

My understanding, from the odd bits and bobs of information I have, is that AWS regions are typically managed somewhat independently.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#296

Post by bigiain »

essayist wrote: post #148
gliderShip wrote: post #107

Earlier quoted context omitted.

Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

From a friend a while back:

https://medium.com/@mpesce/the-great-hack-part-one-attack-70...

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#297

Post by anoncept »

heisenbit wrote: post #103

"Lessons learned from reading post-mortems" http://danluu.com/postmortem-lessons/ is a good place to dig deeper The first graph quoted from a survey paper is a classic fitting the GCE outage well: Initial error --92%--> Incorrect handling of errors explicitly signaled in software

https://mitpress.mit.edu/books/engineering-safer-world is also an excellent resource that more people who care about post-mortems should read.

(As background, the author, MIT Prof. Nancy Leveson, summarizes decades of work in the field, offers groundbreaking new theoretical tools that scale up to some of the world's most complex accidents, and has the experience and evidence to back up their relevance e.g. via work on Therac-25, the Columbia Space Shuttle, and Deepwater Horizon to name just a few...)

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#299

Post by coredog64 »

blackflame7000 wrote: post #224
pavel_lishin wrote: post #206

Earlier quoted context omitted.

Sure, but rayval was talking about a scenario that could happen today. Although looking at the other comments, I think I'm significantly underestimating just how much of modern airliners is dependent on software. The pilots might be able to see that they're heading for disaster, but may not be able to do anything about it.

I know for a fact that there are 3 separate computer systems from 3 separate manufacturers on each Boeing airplane. Auto-pilot always uses the consensus of the 3 machines. It's a pretty far-fetched scenario in real life so I thought we were talking fiction.

Former Boeing software engineer, worked on engineering simulators (where real hardware was in the loop):

There is an idea of triple channel autolanding, wherein the plane uses the consensus of the three autolanding systems. Should no consensus be available, then the pilot is advised that autolanding is not available.

Other than that, any sourcing from different manufacturers is happenstance. 737 avionics are sourced from a different vendor than 747/757/767/777. And different functions can come from different vendors, although vendor consolidation has cut down on that.

I'm not across what happened post 777, as I left Boeing in 1999.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#300

Post by coredog64 »

seanp2k2 wrote: post #179
sriram_sun wrote: post #141

Earlier quoted context omitted.

Car manufacturers conduct recalls all the time. There might be the possibility that a million self-driving cars will be held hostage from a remote control tower simultaneously leading to injury or death to millions. However, in practice, as soon as an issue is discovered, there will be the equivalent of recalls (remote updates) and things like this will be fixed. People who are uncomfortable with self driving cars wi…

Car manufacturers are some of the last people I trust to be doing software updates. The recent Takata airbag recall is an example of the ensuing fecal tornado from large recalls: http://blog.caranddriver.com/massive-takata-airbag-recall-ev... In some cases, people are having to wait months to get new airbags because they just don't have them in stock. In the computer case, would you want to keep driving until they ca…

Allegedly, Honda is offering rental cars to customers who are concerned about their safety and there are no parts available to repair their vehicle.

I say allegedly, because my local Honda dealership told me to pound sand when I asked for a rental car for the day they needed to repair my CRV.

Post reply on HN