An update on Sunday’s service disruption • Hacker News Board

Re: An update on Sunday’s service disruption

#181

Post by mikevin » Tue, Jun 04, 2019, 1:58 PM UTC

grogers wrote: post #168

One of my favorite patterns for updating configuration in-band I learned from Juniper routers. When you enact a new configuration you can have it automatically rollack after some period of time unless you confirm the configuration. Often the pattern is to intentionally have it roll back after a short period (e.g. one minute), then again after a longer period, (e.g. 10 minutes) and the on the last time you make it per…

Networking hardware often has this option and I miss it often when deploying configuration changes to software. Its something a lot more systems could use.

Re: An update on Sunday’s service disruption

#182

Post by xtracto » Tue, Jun 04, 2019, 2:01 PM UTC

harshreality wrote: post #44

> [A] configuration change [...] was intended for a small number of servers in a single region. The configuration was incorrectly applied to a larger number of servers across several neighboring regions, and it caused those regions to stop using more than half of their available network capacity. The network traffic to/from those regions then tried to fit into the remaining network capacity, but it did not. The netwo…

A colleague mine at some point used some Ansible command within AWS which instructed AWS to terminate instances that did not have a specific tag... we were so scared as we saw production instances losing connection one by one, until we realized what was happening.

Fortunately, we had set the important instances to have termination protection. But man, the kind of damage you can do with a single command is huge.

Re: An update on Sunday’s service disruption

#183

Post by stingraycharles » Tue, Jun 04, 2019, 2:02 PM UTC

xtracto wrote: post #179

grogers wrote: post #168

One of my favorite patterns for updating configuration in-band I learned from Juniper routers. When you enact a new configuration you can have it automatically rollack after some period of time unless you confirm the configuration. Often the pattern is to intentionally have it roll back after a short period (e.g. one minute), then again after a longer period, (e.g. 10 minutes) and the on the last time you make it per…

This has been in Windows in the screen resolution configuration option for some time.

I think it has been like this since Windows 3.1, so “some time” might be an understatement.

Re: An update on Sunday’s service disruption

#184

Post by lallysingh » Tue, Jun 04, 2019, 2:06 PM UTC

mcny wrote: post #138

kyrra wrote: post #131

Earlier quoted context omitted.

+1 (I'm a Googler, opinions my own) As someone who has been on oncall for 3 year and done a decent amount of production support, this public doc better explains the cause than the internal one if you aren't well versed in the underlying infra. A config change reduced used network capacity by half, and then things started falling over. And pushing the fix took a while due to the now overloaded network.

What can prevent this from happening in the future? Why was the config change required?

Prioritize configuration traffic so they can fix the problem quicker.

Re: An update on Sunday’s service disruption

#185

Post by lallysingh » Tue, Jun 04, 2019, 2:08 PM UTC

vbsteven wrote: post #63

This reminds me of the time I wanted to test packet loss for a VoIP app and used `tc` to introduce 95% packet loss on the office gateway and because of the packet loss I could not ssh into the box to turn it off... on Google scale.

Running those commands under a container's netem interface helps.

Re: An update on Sunday’s service disruption

#186

Post by jacquesm » Tue, Jun 04, 2019, 2:10 PM UTC

mcny wrote: post #138

kyrra wrote: post #131

Earlier quoted context omitted.

+1 (I'm a Googler, opinions my own) As someone who has been on oncall for 3 year and done a decent amount of production support, this public doc better explains the cause than the internal one if you aren't well versed in the underlying infra. A config change reduced used network capacity by half, and then things started falling over. And pushing the fix took a while due to the now overloaded network.

What can prevent this from happening in the future? Why was the config change required?

Test changes on smaller parts of the network before pushing them to critical parts would be a first guess.

Config changes tend to be nasty in that their implications are often hard to oversee until they have been made, and if the effects preclude you from making another config change then you've just cut off the branch that you were sitting on.

Google is best-in-class when it comes to this stuff, the thing you should take away from this is that if they can mess up everybody does. And that pretty much correlates with my experience to date. This stuff is hard, maybe needlessly so but that does not change the fact that it is hard and that accidents can and will happen. So you plan for things to go wrong when you design your systems. Failure is not only an option, it is the default.

Re: An update on Sunday’s service disruption

#187

Post by endtime » Tue, Jun 04, 2019, 2:13 PM UTC

ineedasername wrote: post #169

dsfyu404ed wrote: post #166

Earlier quoted context omitted.

>Engineers building apps that depend on that know the limitations. As someone who works at a slightly smaller tech company with of similar age with similar infrastructure I assure you this is not the case. Engineers are building things that rely on other things that rely on other things. There's a point where people don't know what their dependencies are. I wouldn't be surprised if nobody actually knew there was cust…

I've never worked in this type of operation, can you shed some light? I would have thought there'd be some type of documentation of the dependency hierarchy for change request checklists. Or are such things not always quite as comprehensive ( or not possible to have such complex interdependencies be comprehensively documented) ?

If you build a new service that uses Spanner, you'd list Spanner as a dependency in your design doc, and maybe even decide to offer an SLO upper-bounded by Spanner's. But you wouldn't list, or even know, the transitive dependencies introduced by using Spanner. You'd more or less have to be the tech lead of the Spanner team to know all the dependencies even one level deep (including whatever 1% experiments they're running and how traffic is selected for them). And even if you ask the tech lead and get a comprehensive answer, it won't be meaningful to anyone reading your launch doc (since they work on, say, Docs, with you), and will be almost immediately out of date.

Google infrastructure is too complicated to know everything. Most of the time, understanding the APIs you need to use (and their quirks and performance tradeoffs and deprecation timelines, etc.) is more than enough work.

> not possible to have such complex interdependencies be comprehensively documented

Yeah, this.

Re: An update on Sunday’s service disruption

#188

Post by Slippery_John » Tue, Jun 04, 2019, 2:22 PM UTC

How did a tool roll out changes to extra regions by accident? Fat-fingering a larger than intended volume in a single region I get, but does their tooling not require explicit opt in for regions? Why does it even allow simultaneous multi-region rollout at all? Is there no auto rollback, or was the failure mode not something that was a considered side-effect of the system?

Re: An update on Sunday’s service disruption

#189

Post by gregschlom » Tue, Jun 04, 2019, 2:23 PM UTC

fragmede wrote: post #180

grogers wrote: post #168

One of my favorite patterns for updating configuration in-band I learned from Juniper routers. When you enact a new configuration you can have it automatically rollack after some period of time unless you confirm the configuration. Often the pattern is to intentionally have it roll back after a short period (e.g. one minute), then again after a longer period, (e.g. 10 minutes) and the on the last time you make it per…

This is called a watchdog timer, and is frequently used in embedded systems where "bare-metal" "OS" upgrades need to occur. You think it's stressful upgrading a router on the other side of the planet, try upgrading the firmware on the Mars Rover! (I've only heard stories.)

That's not really what a whatchdog timer is though.

https://en.m.wikipedia.org/wiki/Watchdog_timer

Re: An update on Sunday’s service disruption

#190

Post by jsty » Tue, Jun 04, 2019, 2:25 PM UTC

gamegod wrote: post #167

markphip wrote: post #149

Is it just me or is this lacking any acknowledgment of the impact it had on GCE and all of the third parties that were impacted by this. They make it sound like a few people could not watch YouTube videos and even fewer people had some email disruption but this outage had a lot more impact than that. As just one example, a huge number of Shopify sites were impacted by this as were I am sure a number of other SaaS bus…

The network became congested, and our networking systems correctly triaged the traffic overload and dropped larger, less latency-sensitive traffic in order to preserve smaller latency-sensitive traffic flows... Overall, YouTube measured a 10% drop in global views during the incident... So what I'm hearing is that while Google Cloud Pub/Sub was down for hours, crippling my SaaS business, Google was prioritizing traffi…

Considering this was seemingly a mostly North America-affecting networking issue, and the 10% reduction in views was global, it doesn't sound like YouTube got much of a priority - quite a lot of the videos that were actually view-able in affected regions during the outage may simply have been served from edge caches.

Disclaimer: no inside knowledge, the above is pure supposition