Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An update on Sunday’s service disruption

cloud.google.com

Post reply on HN
Print view View original

81–90 of 220 posts

Re: An update on Sunday’s service disruption

#81

Post by e12e »

ehsankia wrote: post #39
nodesocket wrote: post #34

> Overall, YouTube measured a 10% drop in global views during the incident, while Google Cloud Storage measured a 30% reduction in traffic. Approximately 1% of active Gmail users had problems with their account; while that is a small fraction of users G suite failed to sync e-mail. My Nest app was completely down via iPhone. Google Home when asked for the weather in Nashville responded with "I can't help with that...…

It didn't say it was a small impact, but that it impacted a small number of users. If you were one of those users, it was high impact for you, but the number of impacted users was small.

They called 1 in hundred a "small fraction". 1% of 1.5 billion users, is not "a small number of users".

Re: An update on Sunday’s service disruption

#83

Post by SquareWheel »

ben_jones wrote: post #49
jsiepkes wrote: post #2

> For most Google users there was little or no visible change to their services—search queries might have been a fraction of a second slower than usual for a few minutes but soon returned to normal, their Gmail continued to operate without a hiccup, and so on. Google probably forgot that some of their own brands are also hosted on their cloud. Like Nest. Basically Nest was down entirely.

GitHub hooks for google cloud builder were also down, meaning if you chose google for deployment automation you were SOL. It wasn’t shown as down under any of the status pages, but it was definitely down for us. I get that outages happen. But having a dishonest status page just plain sucks.

Maybe the server was responding with a 200, but something deeper in the service just wasn't working. I expect these things are complicated and a status page is just an approximation.

Re: An update on Sunday’s service disruption

#84

Post by phs318u »

mrunkel wrote: post #35
snewman wrote: post #29

This is a surprisingly vague postmortem. No timeline, no specific identification of affected regions. And no explanation of why a configuration change that was (apparently) made with a single command required so much effort to undo, or why repair efforts were hampered when (again, apparently) the network was successfully prioritizing high-priority traffic. Even for a public postmortem, this seems pretty weak.

> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ... This is not the post-mortem, that is still to come.

Its the pre-post-mortem.

Re: An update on Sunday’s service disruption

#85

Post by keypusher »

shereadsthenews wrote: post #14
sidibe wrote: post #12

Earlier quoted context omitted.

It is harder to make useful tests for configuration changes than other kinds of changes.

That's not true. Configuration is just another input to a system and the outputs can be checked. There is nothing difficult about testing configs that is not already difficult about testing code.

No, configuration defines the system and is therefore significantly harder to test. Application code can be unit tested and run in staging against simulated real world traffic. Configuration, on the other hand, often differs so much between production and staging that the only real way to test is to roll it out gradually and monitor the results. Just make staging and production identical then right? Easier said than done when your production infrastructure runs most of the worlds internet traffic. There’s a good reason that the last major outages to both AWS and Google were caused by infrastructure configuration changes, and it isn’t that their engineering sucks.

Re: An update on Sunday’s service disruption

#86

Post by laurentl »

phs318u wrote: post #84
mrunkel wrote: post #35

Earlier quoted context omitted.

> With all services restored to normal operation, Google’s engineering teams are now conducting a thorough post-mortem ... This is not the post-mortem, that is still to come.

Its the pre-post-mortem.

The mortem ?

Re: An update on Sunday’s service disruption

#87

Post by theevilsharpie »

vast wrote: post #82

This is very misleading and dodgy. GCP/GCE regions were (reportedly) affected. Gcloud apis were affected even in EU. "others" is a pretty big word here.

Most of our GCE instances are in us-central1 and us-west1, and we saw some intermittently failing health checks and intermittent connectivity to non-GCP resources. Several of my colleagues on the US east coast reported being unable to access their GSuite accounts, but the folks on the US west coast and in eastern Europe seemed to be working fine. In fact, other than watching the Google status updates and our own monitoring systems, most of the conversation was about the fact that Google+ apparently still exists. :P

I don't want to take away from anyone that suffered a significant outage, but the impact did seem to depend on which region you were in, and Google explicitly stated as much in their blog post.

Re: An update on Sunday’s service disruption

#88

Post by laurentl »

singron wrote: post #67
harshreality wrote: post #44

> [A] configuration change [...] was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighboring regions, and it caused those regions to stop using more than half of their available network capacity. The network traffic to/from those regions then tried to fit into the remaining network capacity, but it did not. The netwo…

> The description is vague about what devices ("servers") were misconfigured. "servers" when said by Googlers usually means processes that serve requests, not machines. Hopefully a future postmortem will provide more details. > How would restricted bandwidth utilization on servers cause network congestion... This is a common problem with load balancing if you ever use non-trivial configuration. Imagine you split 100…

> During normal operations, you usually want administrative traffic (e.g. config or executable updates) to be low-priority so it doesn't disrupt production traffic

Honest question: is it not best practice to have an isolated, dedicated management network? I can’t for the life of me understand why a misconfig on the production network should hamper access through the admin network. Unless on Google’s scale it’s not the proper way to design and operate a network ?

Re: An update on Sunday’s service disruption

#89

Post by SpicyLemonZest »

grey-area wrote: post #80
klodolph wrote: post #20

Earlier quoted context omitted.

A large service doesn’t rely on any single piece of hardware. It would take many simultaneous hardware failures to bring down a service. In practice this means a major disaster like a hurricane or fire. The configuration change is just the trigger, though. It’s not that the configuration change is “to blame”. The problem is really that the code doesn’t protect against configurations which can cause outages. After an…

The configuration change is just the trigger, though When there is an outage at a large cloud provider nowadays it's almost always a config change. I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally, separates the control plane, and allow…

Config changes at every major cloud provider I’m familiar with (including Google) satisfy your criteria. They’re testable, incrementally applied, and support easy and immediate rollbacks. And 95% of the time, when someone tries to release a bad config change, those mechanisms prevent or immediately remediate it.

The other 5% are cases like this. How would you discover in advance that “this config will knock over the regional network, but only when deployed at scale” is a potential failure mode? Even if you could, how do you write a test for that?

Re: An update on Sunday’s service disruption

#90

Post by robszumski »

grey-area wrote: post #80
klodolph wrote: post #20

Earlier quoted context omitted.

A large service doesn’t rely on any single piece of hardware. It would take many simultaneous hardware failures to bring down a service. In practice this means a major disaster like a hurricane or fire. The configuration change is just the trigger, though. It’s not that the configuration change is “to blame”. The problem is really that the code doesn’t protect against configurations which can cause outages. After an…

The configuration change is just the trigger, though When there is an outage at a large cloud provider nowadays it's almost always a config change. I don't think it's helpful to treat these as isolated one-offs caused by a bogus configuration. Perhaps what is required is a completely different attitude to config changes, which treats them as testable, applies them incrementally, separates the control plane, and allow…

I am positive that Google has these systems in place. Or at least in many many places where config changes can go wrong. Hopefully they will share what caused it to fail in this circumstance.
Post reply on HN