An update on Sunday’s service disruption • Hacker News Board

Re: An update on Sunday’s service disruption

#21

Post by nodesocket » Tue, Jun 04, 2019, 5:02 AM UTC

milesdyson_phd wrote: post #17

nodesocket wrote: post #11

It feels a little strange and calculated since the outage impacted mostly US based regions, that Google released this update late at night (relative to Central and Eastern time). Wouldn't it make more sense to release it tomorrow, Tuesday at like 11am Eastern (8am Pacific) for full transparency for the affected companies?

I think your time zones are mixed up, but yeah that would make sense. But in the end the people who want/need to know will find it anyway.

Oops, you are right. Edited, but my point stands. :-)

Re: An update on Sunday’s service disruption

#22

Post by shereadsthenews » Tue, Jun 04, 2019, 5:04 AM UTC

reilly3000 wrote: post #9

sdan wrote: post #7

Earlier quoted context omitted.

Even with Kubernetes, you can clearly see what is deploying to what nodes. Not sure what Google's pipeline is, but I would suspect they have some "undo" function to stop the deployment .

I'm guessing it was something lower level than Kubernetes/Borg, since it was able to affect all of their networking bandwidth across multiple regions. ¯\_(ツ)_/¯

The interesting tidbit in here (really the only piece of information at all) is that the outage itself prevented remediation of the outage. That indicates that they might have somehow hosed their DSCP markings such that all traffic was undifferentiated. Large network operators typically implement a "network control" traffic class that trumps all others, for this reason.

Re: An update on Sunday’s service disruption

#23

Post by wilde » Tue, Jun 04, 2019, 5:05 AM UTC

jameslk wrote: post #5

> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?

Configuration often goes through less testing and scrutiny, while simultaneously having a significantly faster deployment process.

Re: An update on Sunday’s service disruption

#24

Post by altmind » Tue, Jun 04, 2019, 5:06 AM UTC

jsiepkes wrote: post #2

> For most Google users there was little or no visible change to their services—search queries might have been a fraction of a second slower than usual for a few minutes but soon returned to normal, their Gmail continued to operate without a hiccup, and so on. Google probably forgot that some of their own brands are also hosted on their cloud. Like Nest. Basically Nest was down entirely.

Most importantly, commerical gsuite was down. Its paid service(with bad, but still a SLA), and some companies worked on sunday. Pretty bad when both corp email and hangouts dont work - no way to communicate remediation steps.

Re: An update on Sunday’s service disruption

#25

Post by nullwasamistake » Tue, Jun 04, 2019, 5:06 AM UTC

gregdoesit wrote: post #18

I wish the Google team shared or could more details on the incident: like the timeline, how long the total outage took and what preventative actions they are taking to fix a similar issue from happening at a systemic level, or mitigation to be substantially faster next time. This update feels like it just shares the root cause at a high level (configuration change) and norms much else.

> Google Cloud Storage measured a 30% reduction in traffic

With things like these the monetary value is so huge their legal team will never allow them to give details. More detail, more chance of lawsuits

Re: An update on Sunday’s service disruption

#26

Post by lclarkmichalek » Tue, Jun 04, 2019, 5:08 AM UTC

sdan wrote: post #7

reilly3000 wrote: post #4

I'm sure there is some code review for the configuration changes, but clearly the engineer(s) and reviewer(s) missed that the scope of the selector it was targeting. I've used Terraform and am learning Pulumi and both provide detailed plans/previews all changes before they are implemented. I wonder how Google's process works for networking configuration. Its so vague its hard to tell what actually happened.

Even with Kubernetes, you can clearly see what is deploying to what nodes. Not sure what Google's pipeline is, but I would suspect they have some "undo" function to stop the deployment .

When you're dealing with this scale of system, the number of config changes, automated or human, can make determining which config the harder issue. Once you've found out what the issue is, you probably also want the revert to go via the normal flows, for fear that your revert could exacerbate the situation. Both of those add time to remediation.

Re: An update on Sunday’s service disruption

#27

Post by flukus » Tue, Jun 04, 2019, 5:12 AM UTC

jameslk wrote: post #5

> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?

Well you can't just change code, it has to get reviewed, it has to go to QA, it has to go to UAT, it has to get signed off in triplicate by all the major stake holders. Configuration changes are easy though, they don't have to go through all these error prevention steps, we can just have our less technical support staff make configuration changes live in production. In fact we'll build them a DSL so we never have to make urgent code changes again...

Re: An update on Sunday’s service disruption

#28

Post by fulafel » Tue, Jun 04, 2019, 5:13 AM UTC

jameslk wrote: post #5

> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?

I don't know if there is a science in "root cause analysis" to pickin the one to put in the headline?

In this case, like most similar ones, it seems obvious that there are many things that conspired together to mess things up. If I got to decide based on the description, I would point the finger at the system properties that caused the 5 hour delay to reconfigure the network capacity.

Re: An update on Sunday’s service disruption

#29

Post by snewman » Tue, Jun 04, 2019, 5:13 AM UTC

This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.

Re: An update on Sunday’s service disruption

#30

Post by paxys » Tue, Jun 04, 2019, 5:15 AM UTC

snewman wrote: post #29

This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.

It's definitely not a postmortem. Seems like just a quick update.