Any other G Suite customers still getting delayed notifications from the outage? I got a batch of 17 last night and another this morning.
An update on Sunday’s service disruption
141–150 of 220 posts
Re: An update on Sunday’s service disruption
#142> However, for users who rely on services homed in the affected regions, the impact was substantial, particularly for services like YouTube or Google Cloud Storage which use large amounts of network bandwidth to operate.
> The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows, much as urgent packages may be couriered by bicycle through even the worst traffic jam.
> Finally, low-bandwidth services like Google Search recorded only a short-lived increase in latency as they switched to serving from unaffected regions, then returned to normal.
I’m pretty sure Nest Thermostats fall in the ultra low bandwidth category. Nobody controlling Nest via devices was able to operate their systems during this outage. Sounds like they better move Nest to the bicycle lane?
I really dislike smarmy “nothing to see here, maybe 10% of YouTube videos were slow” updates. The “1% of Gmail” is even worse, since everyone we know with Gmail was affected. This press release can only be targeting people who don’t use Gmail. (Enterprise cloud buyers, maybe?)
Third party status tracking showed virtually any brand that’s made public splash about hosting on Google Cloud was essentially unreachable for 3 hours. It was amazing to look at the graphs, the correlation was across the board.
Re: An update on Sunday’s service disruption
#143This answered all the questions I had. I was really racking my brain on what one system at Google could go down to cause this much damage, but it makes perfect sense that bandwidth becoming unavailable and everything in the "default" or "bulk" traffic class being dropped would do it.
The real question is whether the fix will be to not reduce bandwidth accidentally, or to upgrade customer traffic to a higher QoS class. It makes sense that internal blob storage is in the lowest "bulk" class. Engineers building apps that depend on that know the limitations. It makes less sense to put customer traffic in that class, though, when you have an SLA to meet for cloud storage. People outside of Google have no idea that different tasks get different network priorities, and don't design their apps with that in mind. (I run all my stuff on AWS and I have no idea what the failure mode when bandwidth is limited for things like EBS or S3. It's probably the same as Google, but I can't design around it because I don't actually know what it looks like.) But, of course, if everything is high priority, nothing is high priority. I imagine that things in the highest traffic class kept working on Sunday, which is a good outcome. If everything were in the highest class, then nothing would work.
(When I worked at Google, I spent a fair amount of time advocating for a higher traffic class for my application's traffic. If my application still exists, I wonder if it was affected, or if the time I spent on that actually paid off.)
Re: An update on Sunday’s service disruption
#144In 50 years, historians will look back on this as the turning point of AI control of humanity, inevitably leading to the point of no return. The brain trust at Google determined that humans are too prone to error to manage their critical data centers so they trained their AI efforts upon the resiliency of their hardware and software systems (i.e. "to prevent human operators from being able to mess it up"). By the tim…
I feel like the capital would be called Muskow.
Re: An update on Sunday’s service disruption
#145> In essence, the root cause of Sunday’s disruption was a configuration change I feel like I hear about config changes breaking these cloud hosts so often it might as well be a meme. Is there a reason why it's usually configurations to blame vs code, hardware, etc?
All infrastructure these days are configurations rather than code. K8s is basically all a set of config files, even a dockerfile is a config file.
Re: An update on Sunday’s service disruption
#146Re: An update on Sunday’s service disruption
#147This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.
they typically post a full post-mortem in the status page of the original incident: https://status.cloud.google.com/incident/compute/19003
Re: An update on Sunday’s service disruption
#148This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.
The lack of transparency makes me want to consider other cloud providers. All providers will have outages -- that's a reality I can live with -- but I will prioritize the ones who are the most forthcoming in their statuses and explanations into failures.
as the other comments here state - wait until they post the full post mortem to draw conclusions. This reads like a statement to Gmail/Youtube customers, not a post-mortem for GCP customers.
Re: An update on Sunday’s service disruption
#149Is it just me or is this lacking any acknowledgment of the impact it had on GCE and all of the third parties that were impacted by this. They make it sound like a few people could not watch YouTube videos and even fewer people had some email disruption but this outage had a lot more impact than that. As just one example, a huge number of Shopify sites were impacted by this as were I am sure a number of other SaaS businesses that use GCE. I realize this is not the complete post mortem but it fails to even acknowledge the full impact of this disruption.
Re: An update on Sunday’s service disruption
#150Earlier quoted context omitted.
Well, one possible remediation for this particular issue would be to separate the control plane for configs from the network the configs control. It appears this bad config stopped them solving the problem in a timely way, which shouldn't really happen. But I don't know the answers, I'm just saying config needs work and we should not pretend the problem lies elsewhere. As the article says, it is the root cause for mo…
So what happens when you need to update the control plane for configs? Do you add another layer?
The ultimate layer will be built from tobacco tins connected with tight wet string.