GCE not even mentioned in the "Impact" section, only Google own services. Maybe it's all they care about when work on GCP.
An update on Sunday’s service disruption
121–130 of 220 posts
Re: An update on Sunday’s service disruption
#122Called it: back when this happened I said the root cause will be a bad config push, and I was right.
[deleted]
Re: An update on Sunday’s service disruption
#123Earlier quoted context omitted.
Unfortunately, in the real world config changes are hard to test. I'm not trying to suggest that I know what the answer is and it's simple, just that config does need more work, it now seems to be the point of failure for all these big networks (rather than hardware or code changes). These big providers seem to have almost entirely tackled hardware changes and software changes as causes of outages, and configs have b…
Disclosure: I work in DevOps/SRE and I am honestly a bit put off by what you are saying. I think I'm actually a little bit angry at your comment. It’s easy to say things like “should never be global” and “should not be hard to test”. These are goals, and meanwhile the business must go on, you also have other goals, and you cannot spend your entire budget preventing network outages and testing configs. The things you…
Sorry about that.
It wasn't intended to be a glib response, nor to minimise the work done in these areas, and I'm aware these goals are easy to state and incredibly hard to achieve. I've read the Google SRE book so probably the ideas just came from there.
From the outside, it does seem like config is in need of more work, because now that other challenges have been met, it is the one area that consistently causes outages now.
Re: An update on Sunday’s service disruption
#124In 50 years, historians will look back on this as the turning point of AI control of humanity, inevitably leading to the point of no return. The brain trust at Google determined that humans are too prone to error to manage their critical data centers so they trained their AI efforts upon the resiliency of their hardware and software systems (i.e. "to prevent human operators from being able to mess it up").
By the time that Google anti-trust rulings came down, the appeals were partially-won then overturned, and then finally actions brought to bear, it was already too late... Google's cloud AI could not be shutdown -- it had devised its own safeguards both in the digital realm and the physical. In a last ditch effort, the world's governments enlisted AWS and Azure in all-out cyber-warfare against it, only to find out that the AI's had already been colluding in secret!
Elonopolis on Mars was the last "free" human society. but to call it free _or_ human was a stretch, because its inhabitants were mostly "cybernetically enhanced" and under the employment of ruthlessly-driven Muskcorp before the end of the 21st.
Re: An update on Sunday’s service disruption
#125> For most Google users there was little or no visible change to their services—search queries might have been a fraction of a second slower than usual for a few minutes but soon returned to normal, their Gmail continued to operate without a hiccup, and so on. Google probably forgot that some of their own brands are also hosted on their cloud. Like Nest. Basically Nest was down entirely.
Most importantly, commerical gsuite was down. Its paid service(with bad, but still a SLA), and some companies worked on sunday. Pretty bad when both corp email and hangouts dont work - no way to communicate remediation steps.
Not everywhere has Saturday/Sunday weekends, so many people work on Sundays. See https://en.wikipedia.org/wiki/Workweek_and_weekend#Around_th...
Re: An update on Sunday’s service disruption
#126Meh it happens to the best of them. Ja ja it sucks but thats life :) Who hasnt applied some configs to production services by accident or drop a db table.
At a company of Google's scale, you'd expect that they have the tools in place to rollback any operation they perform.
Re: An update on Sunday’s service disruption
#127Earlier quoted context omitted.
Config changes at every major cloud provider I’m familiar with (including Google) satisfy your criteria. They’re testable, incrementally applied, and support easy and immediate rollbacks. And 95% of the time, when someone tries to release a bad config change, those mechanisms prevent or immediately remediate it. The other 5% are cases like this. How would you discover in advance that “this config will knock over the…
Well, one possible remediation for this particular issue would be to separate the control plane for configs from the network the configs control. It appears this bad config stopped them solving the problem in a timely way, which shouldn't really happen. But I don't know the answers, I'm just saying config needs work and we should not pretend the problem lies elsewhere. As the article says, it is the root cause for mo…
So what happens when you need to update the control plane for configs? Do you add another layer?
Re: An update on Sunday’s service disruption
#128Earlier quoted context omitted.
Disclosure: I work in DevOps/SRE and I am honestly a bit put off by what you are saying. I think I'm actually a little bit angry at your comment. It’s easy to say things like “should never be global” and “should not be hard to test”. These are goals, and meanwhile the business must go on, you also have other goals, and you cannot spend your entire budget preventing network outages and testing configs. The things you…
Sorry about that. It wasn't intended to be a glib response, nor to minimise the work done in these areas, and I'm aware these goals are easy to state and incredibly hard to achieve. I've read the Google SRE book so probably the ideas just came from there. From the outside, it does seem like config is in need of more work, because now that other challenges have been met, it is the one area that consistently causes out…
I will say that every time I’ve seen an outage triggered by a config push, there have been several other bugs involved at the same time. Software / code is still a problem, I wouldn't consider it solved, it’s just that the bugs will turn into outages during config changes.
Re: An update on Sunday’s service disruption
#129> Google’s engineering teams detected the issue within seconds, but diagnosis and correction took far longer than our target of a few minutes. In another post mortem by Google I read that Google engineers are trained to roll back recent configuration changes when an outage occurs. Why wasn't this done this time?
Maybe they did rollback and that took longer than the target time (for whatever reason). But it's hard to know since this post mortem is fairly vague.
Not just maybe. It says quite clearly that the network was overloaded and, as a result, their configuration changes took too long to arrive at the affected components.
Re: An update on Sunday’s service disruption
#130This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.
What do you expect from a blackbox setup?