Earlier quoted context omitted.
They do SLO by quarter.
Just to clarify, the criteria is using "monthly uptime percentage": https://cloud.google.com/compute/sla
Interesting. I guess quarterly is an internal target.
Post-Mortem for Google Compute Engine’s Global Outage on April 11
351–360 of 368 posts
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#352Earlier quoted context omitted.
> I'm not really sure why I got downvoted for this. Because you have an apparently incredibly simple mental model for the system and so of course tests for it seem simple?
What makes you think software that produces and rolls out configuration files is something complicated? I don't doubt that Google's infrastructure is as complicated and nuanced as it can get. Configuration software just simply isn't. I still don't really see the point you're trying to make here. There isn't enough detail in the two sentences they gave us on the actual cause of the problem to really say much more in a…
What makes you think software that produces and rolls out configuration files is something complicated?
I don't doubt that Google's infrastructure is as complicated and nuanced as it can get. Configuration software just simply isn't.
You literally have no idea what you're talking about.
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#353Earlier quoted context omitted.
> The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. Most people have a greater fear of flying than driving by car although statistically you're far more at risk in a car. One cause of that fear of flying is loss of control; you have to accept placing your life in someone else's hands. With self driving cars suspect lack of control will als…
> Most people have a greater fear of flying than driving by car although statistically you're far more at risk in a car. One cause of that fear of flying is loss of control; you have to accept placing your life in someone else's hands. Really? That sounds counter intuitive. You'd think the reason people are afraid of flying is, because, you know, it's flying. Thirty thousand feet between you and the cold, hard ground…
Maybe not the pilot, but simply being in the cockpit is usually enough to control the fear. Why? First, you can see what is going on, whereas in the cabin you can imagine what might be going on, and think the worst. Second, you can see the pilot is calm. You could watch the flight attendants when in the cabin but that may not be convincing. Third, if you hear or feel something unusual, you can simply ask the captain.
But, since flying in the cockpit isn't available, then what? Get a copy of "SOAR: The Breakthrough Treatment for Fear of Flying" (Amazon editors' 2014 favorite book).
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#354Earlier quoted context omitted.
> The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. Most people have a greater fear of flying than driving by car although statistically you're far more at risk in a car. One cause of that fear of flying is loss of control; you have to accept placing your life in someone else's hands. With self driving cars suspect lack of control will als…
> Most people have a greater fear of flying than driving by car although statistically you're far more at risk in a car. One cause of that fear of flying is loss of control; you have to accept placing your life in someone else's hands. Really? That sounds counter intuitive. You'd think the reason people are afraid of flying is, because, you know, it's flying. Thirty thousand feet between you and the cold, hard ground…
Maybe not the pilot, but simply being in the cockpit is usually enough to control the fear. Why? First, you can see what is going on, whereas in the cabin you can imagine what might be going on, and think the worst. Second, you can see the pilot is calm. You could watch the flight attendants when in the cabin but that may not be convincing. Third, if you hear or feel something unusual, you can simply ask the captain.
But, since flying in the cockpit isn't available, then what? Get a copy of "SOAR: The Breakthrough Treatment for Fear of Flying" (Amazon editors' 2014 favorite book).
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#355Earlier quoted context omitted.
Can't find it with a quick google now, but there was a study on this. Giving people fake controls, even when they knew they were fake, reduced anxiety. Give ambulatory meat some onboard decision-making ability, and it will want to use it.
That's incredible! I would never have guessed. I stand corrected, thank you.
If you want to see the problem practice, here's someone's mother trying out the Tesla self-drive... http://mirror.ninja/sniz
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#356Earlier quoted context omitted.
Preventing the terror plot because the "do not drive over humans" goal overrides the "navigate to preselected target" goal?
my first thought actually was, how would the cars react to road spikes thrown by police? Especially on a highway with barriers on either side. Of course a competent writer would've thrown in a line about how these cars are on run flats at some point... Our only hope is for the scientists in the So-Secret-President-Doesnt-Even-Know Facility to come up with something so crazy it just might work
There are actually already physical barriers throughout the governmental parts of Washington DC to prevent this sort of thing. There are permanent walls, blocks, and poles along the edges of the roads (often well-integrated into the architecture,) and raise-able barriers built into the road surface at intersections. Ain't no cars running over our president!
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#357Earlier quoted context omitted.
> I would personally never buy one of the Google (or any other) self-driving models with no controls. Google cars have the Big Red Button, which shuts off self-driving system and brings the car to a stop. What more controls do you need?
When are you barreling down a highway at 65 miles per hour, turning off the car might not be the best solution.
When you are barreling down a highway at 65 miles per hour and are not paying attention (and you wouldn't, because the car drives itself just fine), giving you controls is much more dangerous (for you and others around you) then not.
Urmson talks about it here: https://youtu.be/Uj-rK8V-rik?t=14m3s
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#358Earlier quoted context omitted.
What makes you think software that produces and rolls out configuration files is something complicated? I don't doubt that Google's infrastructure is as complicated and nuanced as it can get. Configuration software just simply isn't. I still don't really see the point you're trying to make here. There isn't enough detail in the two sentences they gave us on the actual cause of the problem to really say much more in a…
What makes you think software that produces and rolls out configuration files is something complicated? I don't doubt that Google's infrastructure is as complicated and nuanced as it can get. Configuration software just simply isn't. You literally have no idea what you're talking about.
[deleted]
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#359> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ... This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007: > In closing we point out a challenge that we faced in testing our system for which we have no systemat…
Degraded modes of operation is one example of how to visualize masked errors. Another is to trigger an alarm on fallbacks. As a general reflection, many distributed system leave out the cause of their changes and only log actions. Instead of logging "new membership, new members are b,c,d" you are better of logging "node a has not responded to heartbeat in the last 30 seconds, considering it faulty". Following such a…
Great point! More precisely, each state transition in a system should report the old state, the new state, and the triggering event (cause) to a monitoring system (possibly just a log).
Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11
#360I think most people are missing the main failure point: Why does one change propagate automatically to all regions? All this could have been contained if they deployed changes on different regions at different times. That would also help with screwing less your overseas users by running a maintenance at 10am their local time :-)
> These safeguards include a canary step where the configuration is deployed at a single site and that site is verified to still be working correctly, and a progressive rollout which makes changes to only a fraction of sites at a time, so that a novel failure can be caught at an early stage before it becomes widespread. In this event, the canary step correctly identified that the new configuration was unsafe. Crucial…
Waiting a longer time between regional rollouts (so monitoring systems would have time to detect serious failures) would sacrifice deployment latency, but not deployment throughput (assuming deployments can be made in parallel). For continuous deployment, throughput really matters more than latency.