An update on Sunday’s service disruption
cloud.google.com
An update on Sunday’s service disruption
1–10 of 220 posts
Re: An update on Sunday’s service disruption
#2> For most Google users there was little or no visible change to their services—search queries might have been a fraction of a second slower than usual for a few minutes but soon returned to normal, their Gmail continued to operate without a hiccup, and so on.
Google probably forgot that some of their own brands are also hosted on their cloud. Like Nest. Basically Nest was down entirely.
Re: An update on Sunday’s service disruption
#3> For most Google users there was little or no visible change to their services—search queries might have been a fraction of a second slower than usual for a few minutes but soon returned to normal, their Gmail continued to operate without a hiccup, and so on. Google probably forgot that some of their own brands are also hosted on their cloud. Like Nest. Basically Nest was down entirely.
Ok forget the above; coworker pointed out I missed the paragraph below....new rule; only post after 2 coffee in the morning.
Re: An update on Sunday’s service disruption
#4I'm sure there is some code review for the configuration changes, but clearly the engineer(s) and reviewer(s) missed that the scope of the selector it was targeting. I've used Terraform and am learning Pulumi and both provide detailed plans/previews all changes before they are implemented. I wonder how Google's process works for networking configuration. Its so vague its hard to tell what actually happened.
Re: An update on Sunday’s service disruption
#5> In essence, the root cause of Sunday’s disruption was a configuration change
I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?
Re: An update on Sunday’s service disruption
#6The amazing thing is how many times they've had this exact outage or a close relative of it and Ben Treynor still gets to keep his job.
Re: An update on Sunday’s service disruption
#7I'm sure there is some code review for the configuration changes, but clearly the engineer(s) and reviewer(s) missed that the scope of the selector it was targeting. I've used Terraform and am learning Pulumi and both provide detailed plans/previews all changes before they are implemented. I wonder how Google's process works for networking configuration. Its so vague its hard to tell what actually happened.
Even with Kubernetes, you can clearly see what is deploying to what nodes. Not sure what Google's pipeline is, but I would suspect they have some "undo" function to stop the deployment .
Re: An update on Sunday’s service disruption
#8It seems like every time a major cloud vendor goes down its due to a configuration change.
Re: An update on Sunday’s service disruption
#9I'm sure there is some code review for the configuration changes, but clearly the engineer(s) and reviewer(s) missed that the scope of the selector it was targeting. I've used Terraform and am learning Pulumi and both provide detailed plans/previews all changes before they are implemented. I wonder how Google's process works for networking configuration. Its so vague its hard to tell what actually happened.
Even with Kubernetes, you can clearly see what is deploying to what nodes. Not sure what Google's pipeline is, but I would suspect they have some "undo" function to stop the deployment .
I'm guessing it was something lower level than Kubernetes/Borg, since it was able to affect all of their networking bandwidth across multiple regions. ¯\_(ツ)_/¯
Re: An update on Sunday’s service disruption
#10I'm sure there is some code review for the configuration changes, but clearly the engineer(s) and reviewer(s) missed that the scope of the selector it was targeting. I've used Terraform and am learning Pulumi and both provide detailed plans/previews all changes before they are implemented. I wonder how Google's process works for networking configuration. Its so vague its hard to tell what actually happened.
Code is reviewed, but I'm not aware of any companies where terminal commands are reviewed before each execution (though maybe they should be - it seems like every major cloud outage is config related).
It sounds like the change was reviewed and approved but incorrectly pushed.