Earlier quoted context omitted.
What happened?
TBH the linked update is a pretty accurate summary of the postmortem in my opinion.
> linked update
Where?
An update on Sunday’s service disruption
101–110 of 220 posts
Re: An update on Sunday’s service disruption
#102Earlier quoted context omitted.
The configuration change is just the trigger, though When there is an outage at a large cloud provider nowadays it's almost always a config change. I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally, separates the control plane, and allow…
Config changes at every major cloud provider I’m familiar with (including Google) satisfy your criteria. They’re testable, incrementally applied, and support easy and immediate rollbacks. And 95% of the time, when someone tries to release a bad config change, those mechanisms prevent or immediately remediate it. The other 5% are cases like this. How would you discover in advance that “this config will knock over the…
Well, one possible remediation for this particular issue would be to separate the control plane for configs from the network the configs control. It appears this bad config stopped them solving the problem in a timely way, which shouldn't really happen.
But I don't know the answers, I'm just saying config needs work and we should not pretend the problem lies elsewhere. As the article says, it is the root cause for most of these outages now. The parent said:
It’s not that the configuration change is “to blame”.
Which I (and the article) disagree with.
Re: An update on Sunday’s service disruption
#103I wish the Google team shared or could more details on the incident: like the timeline, how long the total outage took and what preventative actions they are taking to fix a similar issue from happening at a systemic level, or mitigation to be substantially faster next time. This update feels like it just shares the root cause at a high level (configuration change) and norms much else.
I found this write up with some metrics/ timeline on Twitter https://lightstep.com/blog/googles-june-2nd-outage-their-sta...
I still don't think it's the full picture. But better than nothing
Re: An update on Sunday’s service disruption
#104> Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes.
In another post mortem by Google I read that Google engineers are trained to roll back recent configuration changes when an outage occurs. Why wasn't this done this time?
Re: An update on Sunday’s service disruption
#105> Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. In another post mortem by Google I read that Google engineers are trained to roll back recent configuration changes when an outage occurs. Why wasn't this done this time?
Maybe they did rollback and that took longer than the target time (for whatever reason). But it's hard to know since this post mortem is fairly vague.
Re: An update on Sunday’s service disruption
#106Earlier quoted context omitted.
Config changes at every major cloud provider I’m familiar with (including Google) satisfy your criteria. They’re testable, incrementally applied, and support easy and immediate rollbacks. And 95% of the time, when someone tries to release a bad config change, those mechanisms prevent or immediately remediate it. The other 5% are cases like this. How would you discover in advance that “this config will knock over the…
Well, one possible remediation for this particular issue would be to separate the control plane for configs from the network the configs control. It appears this bad config stopped them solving the problem in a timely way, which shouldn't really happen. But I don't know the answers, I'm just saying config needs work and we should not pretend the problem lies elsewhere. As the article says, it is the root cause for mo…
Blaming the configuration system for outages is like blaming JIRA for bugs. Google and friends specifically try to ensure, as a matter of policy, that any change which might cause an outage is implemented as a config.
Re: An update on Sunday’s service disruption
#107This reminds me of the time I wanted to test packet loss for a VoIP app and used `tc` to introduce 95% packet loss on the office gateway and because of the packet loss I could not ssh into the box to turn it off... on Google scale.
How did you end up resolving your issue?
The gateway was running on a linux box in a closet somewhere in the building so I was able to go there physically to resolve the issue.
Re: An update on Sunday’s service disruption
#108For a post targeting engineers as audience, the bike example is a bit odd: most readers already know, the problem is instead the lack of technical details in the post.
Re: An update on Sunday’s service disruption
#109Earlier quoted context omitted.
Well, one possible remediation for this particular issue would be to separate the control plane for configs from the network the configs control. It appears this bad config stopped them solving the problem in a timely way, which shouldn't really happen. But I don't know the answers, I'm just saying config needs work and we should not pretend the problem lies elsewhere. As the article says, it is the root cause for mo…
Blaming the configuration system for outages is like blaming JIRA for bugs. Google and friends specifically try to ensure, as a matter of policy, that any change which might cause an outage is implemented as a config.
From the article, the root cause was a configuration change:
"In essence, the root cause of Sunday’s disruption was a configuration change that was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighboring regions...Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage."
I do think we should blame the configuration system, it is clearly not robust enough, not tested enough, and not resilient in case of failure - a bug in the config can bring down the system which manages the config and stop them fixing the original bug.
Re: An update on Sunday’s service disruption
#110Earlier quoted context omitted.
TBH the linked update is a pretty accurate summary of the postmortem in my opinion.
> linked update Where?