Live data from Hacker News

An update on Sunday’s service disruption

cloud.google.com

91–100 of 220 posts

Re: An update on Sunday’s service disruption

#91

It seems like every time a major cloud vendor goes down its due to a configuration change.

They got very good at understanding and dealing with many other sources of failure, such as faulty hardware or broken network links. The systems are explicitly designed to deal with those.

"Build a reliable system out of unreliable parts".

One way to keep the unreliable human in check is to gate all the changes that human would do manually (shell, clicks on buttons etc) through a change management system (usually infrastructure as code) and actuated on the system by pushing some "config".

This is a broader meaning of the word "config"; it captures the whole system, everything that a human would have done to wire it up. The config says which build of your software runs where, it tells your load balancers which traffic to send to which component etc.

When all operations are carried out via configuration pushes, it's no wonder that any human error gets root-caused "config push"

A common way to roll out a new major change is to do a canary deployment, where a component tested so far only in controlled environment gets tested in the real world, but only with a fraction of traffic. The idea is that if the canary component misbehaves it can be quickly rolled back without having cause major disruption.

The deployment of such a canary is a "config" push. But also the instructions to do the "traffic split" to the canary is a config push. The amount of traffic sent to the canary is usually designed to tolerate a fully faulty canary, i.e. the rest of the system that is not running the canary must be able to withstand the full traffic.

When the split is configured incorrectly it can result in "cascading failures" since now dependencies of the overloaded service further amplify the problem. Upstream services issue retries for downstream rpc calls, further increasing the network load.

Now, the outcome can be much more complicated to predict depending on the layer where the change is applied (whether some app workload or the networking infrastructure itself). Some tricks like circuit breakers can mitigate some issues of cascading failures, but eventually you'll also have to push a canary of the circuit breaker itself :-)

I have no idea about the actual outage; I no longer work there. This was just an example to show why "blaming the config push" is practically equivalent to "blame the human".

Configs are just the vectors of change, the same way the fingers of the humans who often take the blame.

Root-causing thus cannot stop there; the end goal is to design a reliable system that can work with unreliable parts, including unreliable changes. It's freaking hard; especially when the changes apply at the level of the system designed to provide the resiliency in the first place.

Re: An update on Sunday’s service disruption

#92

Earlier quoted context omitted.

AWS's post outrage summaries are pretty much the gold standard. e.g. https://aws.amazon.com/message/2329B7/ https://aws.amazon.com/message/41926/ To be fair to Google, they haven't had enough time to perform a detailed autopsy, and some GCP incident summaries have shown meat on the bones e.g. https://status.cloud.google.com/incident/compute/16007 . And balancing the scales, the AWS status page is notorious for showin…

Google also posts detailed postmortems for their more significant outages. Some examples: https://status.cloud.google.com/incident/cloud-networking/18... https://status.cloud.google.com/incident/cloud-pubsub/19001 https://status.cloud.google.com/incident/cloud-networking/18... https://status.cloud.google.com/incident/cloud-networking/18... https://status.cloud.google.com/incident/compute/18012 Given that this was a m…

I hope one of the things Google learns in the post mortem is that the next day summary should clearly include that a full post mortem is coming in the next few days or however long.

Half the people in this thread are overlooking that fact and going into outrage mode.

Re: An update on Sunday’s service disruption

#93

Earlier quoted context omitted.

The status page wasn't up to date, but I don't have any way of backing up that claim. It's certainly isn't tied to any automated system failure reporting -- the status page seems to require manual updates. When minutes turn into hours, and no updates are available on the status page, it certainly doesn't leave me at ease.

We began noticing erratic behavior with some of our GCE instances at around 11:50 Pacific on the day of the outage, and Google posted a notice on their on their status page that GCE was having an outage about 30 minutes later. They also updated the status page every hour (or when they had new information), which is what they said they would do in their status updates. While it sucks that multiple regions malfunctione…

I wasn't able to load a google doc from drive, or my calendar (both g suite), around 11:45 PDT.

Re: An update on Sunday’s service disruption

#94

Earlier quoted context omitted.

A large service doesn’t rely on any single piece of hardware. It would take many simultaneous hardware failures to bring down a service. In practice this means a major disaster like a hurricane or fire. The configuration change is just the trigger, though. It’s not that the configuration change is “to blame”. The problem is really that the code doesn’t protect against configurations which can cause outages. After an…

The configuration change is just the trigger, though When there is an outage at a large cloud provider nowadays it's almost always a config change. I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally, separates the control plane, and allow…

> I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration.

That’s why I said the config change is “just the trigger”. Root cause analysis will generally result in multiple causes for any problem.

> Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally and allows simple rollback.

Google already has that, you can see it in the postmortems for other outages. It’s called canary.

> Code is stored in version control and extensively tested before deployment. Are config changes treated the same way? It certainly doesn't seem like it. Config changes should not be hard to test, hard to diagnose, and hard to rollback.

Unfortunately, in the real world config changes are hard to test. Not always, but often. Working on large deployments has taught me that even with config changes checked in to source control, with automatic canary and gradual rollouts, you will still have outages.

Code doesn’t have 100% test coverage either. Chasing after 100% coverage is a pipe dream.

Re: An update on Sunday’s service disruption

#95

Earlier quoted context omitted.

A large service doesn’t rely on any single piece of hardware. It would take many simultaneous hardware failures to bring down a service. In practice this means a major disaster like a hurricane or fire. The configuration change is just the trigger, though. It’s not that the configuration change is “to blame”. The problem is really that the code doesn’t protect against configurations which can cause outages. After an…

The configuration change is just the trigger, though When there is an outage at a large cloud provider nowadays it's almost always a config change. I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally, separates the control plane, and allow…

> Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally and allows simple rollback.

> Code is stored in version control and extensively tested before deployment. Are config changes treated the same way? It certainly doesn't seem like it. Config changes should not be hard to test, hard to diagnose, and hard to rollback.

Google already do this. The SRE book goes in to details https://landing.google.com/sre/books/

Re: An update on Sunday’s service disruption

#96
post #82

This is very misleading and dodgy. GCP/GCE regions were (reportedly) affected. Gcloud apis were affected even in EU. "others" is a pretty big word here.

Ya, I feel like they didn't really even acknowledge that some regions were completely down for hours. I have a GKE cluster in us-west-2 (Los Angeles) and it was 100% completely inaccessible from the public web, the google cloud console and the google cloud cli.

There was no 'increased latency' and 'partial' outages. It was completely failed for nearly 4 hours. Google console showed a friendly message that I have not yet setup my first GKE cluster and to click here to try it out. They even offered me a $300 credit for first time use.

Re: An update on Sunday’s service disruption

#97
post #67

Earlier quoted context omitted.

> The description is vague about what devices ("servers") were misconfigured. "servers" when said by Googlers usually means processes that serve requests, not machines. Hopefully a future postmortem will provide more details. > How would restricted bandwidth utilization on servers cause network congestion... This is a common problem with load balancing if you ever use non-trivial configuration. Imagine you split 100…

> During normal operations, you usually want administrative traffic (e.g. config or executable updates) to be low-priority so it doesn't disrupt production traffic Honest question: is it not best practice to have an isolated, dedicated management network? I can’t for the life of me understand why a misconfig on the production network should hamper access through the admin network. Unless on Google’s scale it’s not th…

Presumably it's a trade off of complexity against redundancy, and at the scale that google's datacenters run the complexity is too high to make it worthwhile.

Re: An update on Sunday’s service disruption

#98
post #81

Earlier quoted context omitted.

It didn't say it was a small impact, but that it impacted a small number of users. If you were one of those users, it was high impact for you, but the number of impacted users was small.

They called 1 in hundred a "small fraction". 1% of 1.5 billion users, is not "a small number of users".

“while that is a small fraction of users, it still represents millions of users who couldn’t receive or send email. As Gmail users ourselves, we know how disruptive losing an essential tool can be!”

Re: An update on Sunday’s service disruption

#100

Earlier quoted context omitted.

The configuration change is just the trigger, though When there is an outage at a large cloud provider nowadays it's almost always a config change. I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally, separates the control plane, and allow…

> I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. That’s why I said the config change is “just the trigger”. Root cause analysis will generally result in multiple causes for any problem. > Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally and allows simple rollback. Google already has…

Unfortunately, in the real world config changes are hard to test.

I'm not trying to suggest that I know what the answer is and it's simple, just that config does need more work, it now seems to be the point of failure for all these big networks (rather than hardware or code changes). These big providers seem to have almost entirely tackled hardware changes and software changes as causes of outages, and configs have been exposed as the new point of failure. That will require rethinking how configs are managed and how they are applied. I'm not talking about 100% test coverage, but failure recovery.

The article does suggest that config was the root cause:

In essence, the root cause of Sunday’s disruption was a configuration change that was intended for a small number of servers in a single region

What I'm suggesting is that what google (and Amazon) has for configs is not good enough, that the root cause of this outage was in fact a config change (like all the others), and that what is required is a rethink of configs which recognises that they need an entirely separate control plane, should never be global, should not be hard to test etc.

Clearly here, since the bad config was able to stop them actually fixing the problem, they need to rethink how their configs are applied somehow. As this keeps happening with different config changes I'd suggest this is not a one-off isolated problem but a symptom of a broader failure to tackle the fragile nature of current config systems.

Post reply on HN