Earlier quoted context omitted.
The postmortem is currently only visible to Googlers.
What happened?
TBH the linked update is a pretty accurate summary of the postmortem in my opinion.
An update on Sunday’s service disruption
61–70 of 220 posts
Re: An update on Sunday’s service disruption
#62> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?
It's common practice in many organizations to effectively deploy new code with configuration changes. E.g. you write new code that is initially disabled then enable it with a configuration change. Since the code deploy didn't fail, you get a false confidence in the code and the configuration change seems like a safe "flag flip".
I would phrase this a bit differently:
Binary version changes are a special case of configuration change that we (swes?) are particularly adept at managing reliably and safely.
But there are lots of other config changes that are potentially dangerous, and that we aren't as good at doing safely.
Re: An update on Sunday’s service disruption
#63This reminds me of the time I wanted to test packet loss for a VoIP app and used `tc` to introduce 95% packet loss on the office gateway and because of the packet loss I could not ssh into the box to turn it off... on Google scale.
Re: An update on Sunday’s service disruption
#64Earlier quoted context omitted.
The lack of transparency makes me want to consider other cloud providers. All providers will have outages -- that's a reality I can live with -- but I will prioritize the ones who are the most forthcoming in their statuses and explanations into failures.
Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened. If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?
The status page wasn't up to date, but I don't have any way of backing up that claim. It's certainly isn't tied to any automated system failure reporting -- the status page seems to require manual updates. When minutes turn into hours, and no updates are available on the status page, it certainly doesn't leave me at ease.
Re: An update on Sunday’s service disruption
#65This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.
But it's not a post mortem? Since the article states:
> Next Steps
> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem to ensure we understand all the contributing factors
Re: An update on Sunday’s service disruption
#66Earlier quoted context omitted.
It is harder to make useful tests for configuration changes than other kinds of changes.
That's not true. Configuration is just another input to a system and the outputs can be checked. There is nothing difficult about testing configs that is not already difficult about testing code.
There is: many configuration changes are, by nature, global.
Code changes can be isolated and unit tested. Config changes often can't be.
You can still canary them, usually, but you lose some protection.
Re: An update on Sunday’s service disruption
#67> [A] configuration change [...] was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighboring regions, and it caused those regions to stop using more than half of their available network capacity. The network traffic to/from those regions then tried to fit into the remaining network capacity, but it did not. The netwo…
> The description is vague about what devices ("servers") were misconfigured.
"servers" when said by Googlers usually means processes that serve requests, not machines. Hopefully a future postmortem will provide more details.
> How would restricted bandwidth utilization on servers cause network congestion...
This is a common problem with load balancing if you ever use non-trivial configuration. Imagine you split 100 qps of traffic between equally sized pods A and B. If each pod has an actual capacity of 60 qps and received 50 qps, then everything is fine. However, if you configure your load balancer not to send more than 10 qps to A, then it has to send the remaining 90 qps to B. Now B is actually overloaded by 50%. Using automatic utilization based load balancing can prevent this in some cases, but it can also cause it if utilization isn't reported accurately.
> Someone forgot to classify management traffic as high-priority? Oops.
I have some sympathy. During normal operations, you usually want administrative traffic (e.g. config or executable updates) to be low-priority so it doesn't disrupt production traffic. If you have extreme foresight, maybe you ignored that temptation or built in an escape hatch for emergencies. However, with a complicated layered infrastructure, it's very difficult to be sure that all network communication has the appropriate network priority, and you usually don't find out until a situation like this.
Re: An update on Sunday’s service disruption
#68Earlier quoted context omitted.
Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened. If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?
The status page wasn't up to date, but I don't have any way of backing up that claim. It's certainly isn't tied to any automated system failure reporting -- the status page seems to require manual updates. When minutes turn into hours, and no updates are available on the status page, it certainly doesn't leave me at ease.
We began noticing erratic behavior with some of our GCE instances at around 11:50 Pacific on the day of the outage, and Google posted a notice on their on their status page that GCE was having an outage about 30 minutes later. They also updated the status page every hour (or when they had new information), which is what they said they would do in their status updates.
While it sucks that multiple regions malfunctioned simultaneously for several hours, I can't really fault them for their communication about the issue.
Re: An update on Sunday’s service disruption
#69Earlier quoted context omitted.
The lack of transparency makes me want to consider other cloud providers. All providers will have outages -- that's a reality I can live with -- but I will prioritize the ones who are the most forthcoming in their statuses and explanations into failures.
Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened. If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?
AWS's post outrage summaries are pretty much the gold standard.
e.g.
https://aws.amazon.com/message/2329B7/
https://aws.amazon.com/message/41926/
To be fair to Google, they haven't had enough time to perform a detailed autopsy, and some GCP incident summaries have shown meat on the bones e.g. https://status.cloud.google.com/incident/compute/16007. And balancing the scales, the AWS status page is notorious for showing green when things are ... not so verdant.
I have seen full internal outage tickets and the volume of detail is unsurprisingly vast, and boiling it down into summaries - both internal and external - without whitewashing, without emotion, to capture an honest and coherent narration of all the relevant events and all the useful forward learnings is an epic task for even a skilled technical writer and/or principal engineer. You don't get to rest just because services are up, some folks at Google will have a sleep deficit this week.
Re: An update on Sunday’s service disruption
#70I would be interested in knowing what the configuration change exactly was. Which flags did it turn on/off.