Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on Sunday’s service disruption

cloud.google.com

Post reply on HN
Print view View original

131–140 of 220 posts

Re: An update on Sunday’s service disruption

#131

Post by kyrra »

tylerhou wrote: post #61
echelon wrote: post #50

Earlier quoted context omitted.

What happened?

TBH the linked update is a pretty accurate summary of the postmortem in my opinion.

+1

(I'm a Googler, opinions my own) As someone who has been on oncall for 3 year and done a decent amount of production support, this public doc better explains the cause than the internal one if you aren't well versed in the underlying infra.

A config change reduced used network capacity by half, and then things started falling over. And pushing the fix took a while due to the now overloaded network.

Re: An update on Sunday’s service disruption

#132

Post by grey-area »

buckminster wrote: post #127
grey-area wrote: post #102

Earlier quoted context omitted.

Well, one possible remediation for this particular issue would be to separate the control plane for configs from the network the configs control. It appears this bad config stopped them solving the problem in a timely way, which shouldn't really happen. But I don't know the answers, I'm just saying config needs work and we should not pretend the problem lies elsewhere. As the article says, it is the root cause for mo…

So what happens when you need to update the control plane for configs? Do you add another layer?

Who will configure the configurators?

Hopefully this layer would be far more stable and very infrequently touched.

Re: An update on Sunday’s service disruption

#134

Post by dragon_greens »

phs318u wrote: post #84
mrunkel wrote: post #35

Earlier quoted context omitted.

> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ... This is not the post-mortem, that is still to come.

Its the pre-post-mortem.

Pre-post-erous mortem?

Re: An update on Sunday’s service disruption

#135

Post by q3k »

grey-area wrote: post #132
buckminster wrote: post #127

Earlier quoted context omitted.

So what happens when you need to update the control plane for configs? Do you add another layer?

Who will configure the configurators? Hopefully this layer would be far more stable and very infrequently touched.

And something that's infrequently touched will likely be poorly understood and engineers will not have much knowledge on how to fix things fast when they break.

Re: An update on Sunday’s service disruption

#137

Post by killjoywashere »

awalton wrote: post #136
laurentl wrote: post #86

Earlier quoted context omitted.

The mortem ?

Perimortem?

In pathology we call issue a "preliminary autopsy report" within the first 48 hours and a "final autopsy report" within 30-60 days. A final report may be followed by additional addenda (new information, e.g. toxicology reports) and amendments (significant changes to the original report, e.g. "I was wrong")

Re: An update on Sunday’s service disruption

#138

Post by mcny »

kyrra wrote: post #131
tylerhou wrote: post #61

Earlier quoted context omitted.

TBH the linked update is a pretty accurate summary of the postmortem in my opinion.

+1 (I'm a Googler, opinions my own) As someone who has been on oncall for 3 year and done a decent amount of production support, this public doc better explains the cause than the internal one if you aren't well versed in the underlying infra. A config change reduced used network capacity by half, and then things started falling over. And pushing the fix took a while due to the now overloaded network.

What can prevent this from happening in the future? Why was the config change required?

Re: An update on Sunday’s service disruption

#139

Post by navaati »

aeijdenberg wrote: post #79
reilly3000 wrote: post #4

I'm sure there is some code review for the configuration changes, but clearly the engineer(s) and reviewer(s) missed that the scope of the selector it was targeting. I've used Terraform and am learning Pulumi and both provide detailed plans/previews all changes before they are implemented. I wonder how Google's process works for networking configuration. Its so vague its hard to tell what actually happened.

We use Terraform a lot too - and most of the time it's great, but not infallible. Our team managed to screw-up some pretty major DNS due to a valid terraform plan that looked OK, but in reality then deleted a bunch of records, before failing (for some reason I can't remember) before it could create new ones. And of course, we forgot that although we had shortened TTL on our records, the TTL on the parent records that…

> but in reality then deleted a bunch of records, before failing […] before it could create new ones.

    lifecycle {
      create_before_destroy = true
    }
may be your friend :) (not sure if applicable though)

Re: An update on Sunday’s service disruption

#140

Post by obstacle1 »

perlgeek wrote: post #104

> Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. In another post mortem by Google I read that Google engineers are trained to roll back recent configuration changes when an outage occurs. Why wasn't this done this time?

The literal next paragraph:

> Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage

Post reply on HN