Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on Sunday’s service disruption

cloud.google.com

Post reply on HN
Print view View original

71–80 of 220 posts

Re: An update on Sunday’s service disruption

#71

Post by theevilsharpie »

JMTQp8lwXL wrote: post #41
mrunkel wrote: post #35

Earlier quoted context omitted.

> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ... This is not the post-mortem, that is still to come.

The article leaves it to the reader to decide whether or not we will be updated with the results of that post-mortem.

While this article leaves it ambiguous, the incident status report[1] makes it clear that customers can expect an actual postmortem:

"We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a detailed report of this incident once we have completed our internal investigation. This detailed report will contain information regarding SLA credits."

[1] https://status.cloud.google.com/incident/cloud-networking/19...

Re: An update on Sunday’s service disruption

#72

Post by bowmessage »

vbsteven wrote: post #63

This reminds me of the time I wanted to test packet loss for a VoIP app and used `tc` to introduce 95% packet loss on the office gateway and because of the packet loss I could not ssh into the box to turn it off... on Google scale.

How did you end up resolving your issue?

Re: An update on Sunday’s service disruption

#73

Post by zedgerman »

thundergolfer wrote: post #48
eyjafjallajokul wrote: post #19

Earlier quoted context omitted.

We do at AWS. Not all commands though, but most commands that we audit and find are dangerous. See a similar outage in S3 from 2 years ago - https://aws.amazon.com/message/41926/

How do this code-review of cmds work? Does the command get saved to a file, and then that file is reviewed like regular source-code, and then when it is approved the cmd is copy-pasted back to the terminal and run? That above seems pretty clunky, so it's very likely not what happens.

I’ve seen scripts get checked in and deployed just like you would a new service (code). Same Code Review process and same release pipeline.

In this particular case, commands that were run on a Production machine were by-design limited to what they can do and affect (mostly just the physical host they’re run on or a few hosts in the logical group of hosts they belong to).

Re: An update on Sunday’s service disruption

#74

Post by theevilsharpie »

inopinatus wrote: post #69
theevilsharpie wrote: post #58

Earlier quoted context omitted.

Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened. If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?

AWS's post outrage summaries are pretty much the gold standard. e.g. https://aws.amazon.com/message/2329B7/ https://aws.amazon.com/message/41926/ To be fair to Google, they haven't had enough time to perform a detailed autopsy, and some GCP incident summaries have shown meat on the bones e.g. https://status.cloud.google.com/incident/compute/16007 . And balancing the scales, the AWS status page is notorious for showin…

Google also posts detailed postmortems for their more significant outages.

Some examples:

https://status.cloud.google.com/incident/cloud-networking/18...

https://status.cloud.google.com/incident/cloud-pubsub/19001

https://status.cloud.google.com/incident/cloud-networking/18...

https://status.cloud.google.com/incident/cloud-networking/18...

https://status.cloud.google.com/incident/compute/18012

Given that this was a multi-region outage that lasted several hours and impacted a substantial number of services, I'd expect a detailed postmortem to follow.

Re: An update on Sunday’s service disruption

#75

Post by xenospn »

bowmessage wrote: post #72
vbsteven wrote: post #63

This reminds me of the time I wanted to test packet loss for a VoIP app and used `tc` to introduce 95% packet loss on the office gateway and because of the packet loss I could not ssh into the box to turn it off... on Google scale.

How did you end up resolving your issue?

"Have you tried unplugging it and plugging it back in again?"

Re: An update on Sunday’s service disruption

#76

Post by EGreg »

inopinatus wrote: post #69
theevilsharpie wrote: post #58

Earlier quoted context omitted.

Google kept their status page up to date as the outage was progressing, and now (the day after the outage), they've provided an apology and a preliminary explanation of what happened. If that's not sufficient, what more are you looking for, and what other large cloud providers consistently meet that standard?

AWS's post outrage summaries are pretty much the gold standard. e.g. https://aws.amazon.com/message/2329B7/ https://aws.amazon.com/message/41926/ To be fair to Google, they haven't had enough time to perform a detailed autopsy, and some GCP incident summaries have shown meat on the bones e.g. https://status.cloud.google.com/incident/compute/16007 . And balancing the scales, the AWS status page is notorious for showin…

“post outrage summaries” hehe

Re: An update on Sunday’s service disruption

#79

Post by aeijdenberg »

reilly3000 wrote: post #4

I'm sure there is some code review for the configuration changes, but clearly the engineer(s) and reviewer(s) missed that the scope of the selector it was targeting. I've used Terraform and am learning Pulumi and both provide detailed plans/previews all changes before they are implemented. I wonder how Google's process works for networking configuration. Its so vague its hard to tell what actually happened.

We use Terraform a lot too - and most of the time it's great, but not infallible.

Our team managed to screw-up some pretty major DNS due to a valid terraform plan that looked OK, but in reality then deleted a bunch of records, before failing (for some reason I can't remember) before it could create new ones.

And of course, we forgot that although we had shortened TTL on our records, the TTL on the parent records that I think get hit when no records are found were much longer, so we had a real bad afternoon. :)

Re: An update on Sunday’s service disruption

#80

Post by grey-area »

klodolph wrote: post #20
jameslk wrote: post #5

> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?

A large service doesn’t rely on any single piece of hardware. It would take many simultaneous hardware failures to bring down a service. In practice this means a major disaster like a hurricane or fire. The configuration change is just the trigger, though. It’s not that the configuration change is “to blame”. The problem is really that the code doesn’t protect against configurations which can cause outages. After an…

The configuration change is just the trigger, though

When there is an outage at a large cloud provider nowadays it's almost always a config change. I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration.

Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally, separates the control plane, and allows simple rollback.

Code is stored in version control and extensively tested before deployment. Are config changes treated the same way? It certainly doesn't seem like it. Config changes should not be hard to test, hard to diagnose, and hard to rollback.

Post reply on HN