Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

261–270 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#261

Post by pmarreck »

mjibson wrote: post #27
cjbprime wrote: post #17

It looks like there were at least three catastrophic bugs present: 1. Evaluated a configuration change before the change had finished syncing across all configuration files, resulting in rejecting the change. 2. So it tried to reject the change, but actually just deleted everything instead. 3. Something was supposed to catch changes that break everything, and it detected that everything was broken but its attempt to…

I'm attempting to even imagine how one would build a useful way to test this. Would they have to have a secondary, world-wide datacenter network with all their various services behind it?

You fake out the connection with a faker object and give that to the code that wants to communicate to the network, and it returns streamed, deterministic data that would have been expected from the actual network, given deterministic inputs. The test uses the fake; the production code gets given the real object.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#262

Post by pmarreck »

teraflop wrote: post #31

> There are a number of lessons to be learned from this event -- for example, that the safeguard of a progressive rollout can be undone by a system designed to mask partial failures -- ... This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007: > In closing we point out a challenge that we faced in testing our system for which we have no systemat…

> So, has anyone managed to make progress toward a "systematic solution" in the last 9 years?

Allow me to introduce you to the fantastic and battle-tested http://learnyousomeerlang.com/what-is-otp , preferably utilized (IMHO) via http://elixir-lang.org/

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#263

Post by delroth »

swills wrote: post #139

The thing that stood out for me was: "...team...worked in shifts overnight..."

(Usual disclaimer: I speak for myself, not for my employer, etc.)

The team in charge of solving this particular problem is located in two sites in two different timezones. This is true of most critical SRE teams at Google, and it is precisely to be able to have 24h coverage in these time sensitive situations.

In the 2+ years I have spent in SRE I have never heard of a single instance of an SRE being asked or even encouraged to stay after hours (let alone overnight) for incident remediation. There is quite a lot of emphasis being put on work/life balance.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#264

Post by DanielDent »

erichocean wrote: post #163
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Are BGP updates for Google's own router configurations really so frequent that they can't pay an engineer to at least monitor the propagation of configuration changes? In this case, a human would have instantly seen that the update was a) rejected (as explained in the postmortem), and b) holy shit, WHY DID THE ROUTER CHANGE ITS OWN CONFIGURATION TO BLOW AWAY ALL OF THE GCE ROUTES!?! I'm all for automation, but WTF? I…

Any sufficiently large system quickly reaches a point where a human has difficulty tracking what the system should look like.

Google has at least tens of data center locations, each of which will have multiple physical failure domains.

There are also many discontiguous routes being announced at all of their network PoPs. They have substantially more PoPs than data centers.

It very quickly gets too much to reasonably expect people to be able to keep track of what the system should look like, let alone grasping what it does look like.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#265

Post by NetStrikeForce »

I think most people are missing the main failure point: Why does one change propagate automatically to all regions?

All this could have been contained if they deployed changes on different regions at different times. That would also help with screwing less your overseas users by running a maintenance at 10am their local time :-)

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#266

Post by __jal »

nothrabannosir wrote: post #255
harryf wrote: post #252

Earlier quoted context omitted.

> The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. Most people have a greater fear of flying than driving by car although statistically you're far more at risk in a car. One cause of that fear of flying is loss of control; you have to accept placing your life in someone else's hands. With self driving cars suspect lack of control will als…

> Most people have a greater fear of flying than driving by car although statistically you're far more at risk in a car. One cause of that fear of flying is loss of control; you have to accept placing your life in someone else's hands. Really? That sounds counter intuitive. You'd think the reason people are afraid of flying is, because, you know, it's flying. Thirty thousand feet between you and the cold, hard ground…

Can't find it with a quick google now, but there was a study on this. Giving people fake controls, even when they knew they were fake, reduced anxiety.

Give ambulatory meat some onboard decision-making ability, and it will want to use it.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#267

Post by aiiane »

NetStrikeForce wrote: post #265

I think most people are missing the main failure point: Why does one change propagate automatically to all regions? All this could have been contained if they deployed changes on different regions at different times. That would also help with screwing less your overseas users by running a maintenance at 10am their local time :-)

> These safeguards include a canary step where the configuration is deployed at a single site and that site is verified to still be working correctly, and a progressive rollout which makes changes to only a fraction of sites at a time, so that a novel failure can be caught at an early stage before it becomes widespread. In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step’s conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.

The system does do progressive rollouts, which are essentially what you are referring to (albeit perhaps at a different pace). The number of changes being rolled out means that it's not really feasible to hand roll out configurations to different regions, so the checks are automated. In this case, the automated checks failed as well.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#268

Post by aiiane »

DanielDent wrote: post #257

My post yesterday seems even more relevant today: https://news.ycombinator.com/item?id=11477552 It's a shame it's not easier or more common for people to create clones of (most|all) of their infrastructure for testing purposes. Something like half of outages are caused by configuration oopsies. If you accept that configuration is code, then you also come to the following disturbing conclusion: the usual test environm…

The main issue there is that "environments" are defined by configuration, so if you try to set up a configuration test environment, you run into a direct logical impass: either your configs are production configs, and thus not a separate environment, or they're different from production configs, and thus may provide different test results from production.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#269

Post by dsl »

erichocean wrote: post #163
brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

Are BGP updates for Google's own router configurations really so frequent that they can't pay an engineer to at least monitor the propagation of configuration changes? In this case, a human would have instantly seen that the update was a) rejected (as explained in the postmortem), and b) holy shit, WHY DID THE ROUTER CHANGE ITS OWN CONFIGURATION TO BLOW AWAY ALL OF THE GCE ROUTES!?! I'm all for automation, but WTF? I…

First of all, BGP is core to Google's load balancing architecture. So within a single datacenter you probably have at least a few dozen devices down stream from each edge router.

Secondly, I'm seeing just shy of 500 individual prefixes, 282 directly connected peers (other networks), and a presence at over 100 physical internet exchanges, just for one of Google's four ASes.

Would you be able to read over that configuration and tell me if it has errors?

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#270

Post by DickingAround »

Afforess wrote: post #154
essayist wrote: post #148

Earlier quoted context omitted.

I've been itching to release this terror movie plot into the wild: It's 2025 and more than 10% of the cars on the road in the US are self-driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation. But then, at 430pm, highway patrol u…

I'm afraid Daemon (novel) beat you to the punch. It's an excellent novel, about fairly similar situations. http://www.goodreads.com/book/show/6665847-daemon

Daemon is in that area but I'd for sure also enjoy the original proposal.
Post reply on HN