Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-Mortem for Google Compute Engine’s Global Outage on April 11

status.cloud.google.com

Post reply on HN
Print view View original

281–290 of 368 posts

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#281

Post by advisedwang »

ndesaulniers wrote: post #145

At Google, they do these really awesome post-mortems when there's a major failure. It provides a point of reflection, and are usually well written entertaining reads. Didn't know they made (some?) public. They're a good learning exercise writing one, and is more of a learning exercise than a punishment.

It's worth noting that the publicly posted postmortem is not the same as the internal postmortems (which include much more detail, specific action items, timelines etc). The SRE book (https://landing.google.com/sre/book.html) has a whole chapter on our internal postmortems, which is probably a better learning exercise in how to write one.

Source: I work on the team that writes these external postmortems.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#282

Post by lugg »

brianwawok wrote: post #2

This is a very good Post-Mortem. As I assumed it was kind of a corner case bug meet corner case bug met corner case bug. This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them? Making software is hard....

This wasn't an edge case. It was two bugs in two sections of code both designed to recover from a serious problem. It sounds like both sections of code were not tested properly at the very least.

Sounds to me like someone just didn't bother to test the failsafe part of the code.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#283

Post by NetStrikeForce »

aiiane wrote: post #267
NetStrikeForce wrote: post #265

I think most people are missing the main failure point: Why does one change propagate automatically to all regions? All this could have been contained if they deployed changes on different regions at different times. That would also help with screwing less your overseas users by running a maintenance at 10am their local time :-)

> These safeguards include a canary step where the configuration is deployed at a single site and that site is verified to still be working correctly, and a progressive rollout which makes changes to only a fraction of sites at a time, so that a novel failure can be caught at an early stage before it becomes widespread. In this event, the canary step correctly identified that the new configuration was unsafe. Crucial…

I'm not sure you really understand what I've tried to say, but it's probably my fault because of my poor grasp of the English language.

You are just confirming my previous comment. Your rollouts are automated, so pushing a change automatically configures every region, instead of configuring just one and maybe waiting for a prudential time in human scale before the next one because, surprise!, shit happens.

I understand your colleagues probably make lots of changes, but if that introduces risks of global outages IMHO you should reconsider your strategy.

And I'm not sure why you downvoted my previous comment. It's a perfectly valid observation, based on the published information.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#284

Post by jpatokal »

ndesaulniers wrote: post #145

At Google, they do these really awesome post-mortems when there's a major failure. It provides a point of reflection, and are usually well written entertaining reads. Didn't know they made (some?) public. They're a good learning exercise writing one, and is more of a learning exercise than a punishment.

Google publishes a public incident report for all service outages (code red) in the Cloud status dashboard. You can see some in the History page: https://status.cloud.google.com/summary

Sample: https://status.cloud.google.com/incident/appengine/16002

Note that the length of the report tends to correlate with the severity of the outages and that disruptions (code orange) disruptions do not get reports.

Disclaimer: I work in Cloud Support and write some of these.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#286

Post by oconnor663 »

joering2 wrote: post #280
Gravityloss wrote: post #181

I'm waiting for the time when they push over the air updates to airplanes in flight. "You can fly safely, we have canaries and staged deployment" A year forward: "Unfortunately because the canary verification as well as the staged deployment code was broken, instead of one crash and 300 dead, an update was pushed to all aircraft, which subsequently caused them to crash, killing 70,000 people." I'm not 100% sure why t…

Yeah ,the part with canary code rub me the wrong way too. These safeguards include a canary step where the configuration is deployed at a single site and that site is verified to still be working correctly This sounds very unprofessional imho. "Touch this cable to see if there is electricity running" sort of thing. Is that really how its should be done?

If you're doing electrical work, eventually you're going to have to touch the cable!

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#287

Post by kijiki »

nickysielicki wrote: post #86

What does Google use for BGP? Quagga, OpenBGPD, BIRD, their own? Also, does anyone have a link to statistics on global BGP software usage? I'm curious what the marketshare looks like.

Google has contributed ISIS and BGP code to Quagga in the past, as well as funding some testing at the OSRF. Presumably they use it in at least some parts of their operations.

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#288

Post by sk5t »

seanp2k2 wrote: post #175
rayval wrote: post #157

Earlier quoted context omitted.

Cool plot line, I'd go see that movie. A related scenario, one that theoretically could happen today, is hacking into commercial airliners auto-pilot systems, and directing dozens of flights onto a target. Set aside the fantasy movie plot angle, how realistic is this today? Is it any more or less plausible than the millions of cars scenario? If people are truly concerned about the car scenario, shouldn't they be worr…

In the airplane case, it's possible today: https://m.youtube.com/watch?v=CXv1j3GbgLk And https://m.youtube.com/watch?v=Uy3nXXZgqmg TL;DR you simulate a bunch of other planes in close proximity and the auto-pilot freaks out and tries to avoid them. As the second talk explains, the pilots would definitely notice and switch autopilot off. This is why IMO it's very important to not take ultimate control away from humans…

A minor point, but the electronic accelerator control in autos is called "throttle-by-wire."

Re: Post-Mortem for Google Compute Engine’s Global Outage on April 11

#290

Post by grogers »

joering2 wrote: post #280
Gravityloss wrote: post #181

I'm waiting for the time when they push over the air updates to airplanes in flight. "You can fly safely, we have canaries and staged deployment" A year forward: "Unfortunately because the canary verification as well as the staged deployment code was broken, instead of one crash and 300 dead, an update was pushed to all aircraft, which subsequently caused them to crash, killing 70,000 people." I'm not 100% sure why t…

Yeah ,the part with canary code rub me the wrong way too. These safeguards include a canary step where the configuration is deployed at a single site and that site is verified to still be working correctly This sounds very unprofessional imho. "Touch this cable to see if there is electricity running" sort of thing. Is that really how its should be done?

Think of the canary as the last line of defense, not the first. You always aspire to deploy zero bugs into production, through good testing and other QA. But if a problem happens, you want to limit the impact as much as possible. Affecting one site isn't great, but there is enough redundancy that overall service should be unaffected.
Post reply on HN