Google Cloud Networking Incident Postmortem • Hacker News Board

Re: Google Cloud Networking Incident Postmortem

#161

Post by log_n » Fri, Jun 07, 2019, 2:58 PM UTC

karlding wrote: post #95

londons_explore wrote: post #82

Earlier quoted context omitted.

It isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of the network control plane.

I don't work as an SRE, but isn't that covered by providing engineers physical access to secure facilities in the absolute worst case? The article states: > The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in prior…

You don't have to go that far. You could also have automated roll-back of individual servers if they sense something is off, for instance.

Another alternative is low bandwidth flag based roll-backs (for instances such as this where the network is congested but not completely lost).

Re: Google Cloud Networking Incident Postmortem

#162

Post by ethbro » Fri, Jun 07, 2019, 3:10 PM UTC

di4na wrote: post #147

tetha wrote: post #98

Earlier quoted context omitted.

Someone at our place recently commented that the ops team during an incident strongly feels like NASA mission control in critical moments[1]. I wanted to protest, but that's surprisingly accurate. > And if anyone is screaming or gets emotional, they should not be in that room. If someone starts yelling around in my incident war room for no reason, they get thrown out. I'm a calm and quiet person, but bugging around d…

It is not surprising at all. Mission Control was forged in the fire (literally for Apollo 1) and they are one of the most visible "incident team" we know about. I highly advise to read Gene Kranz memoirs "Failure is not an Option" if you work in that kind of environment.

I heard recently that he never said that.

Apparently it was mentioned when the Apollo 13 script writers were gathering stories at NASA, they liked it, and then gave it to the Kranz character.

Who then decided, "Hey, if everyone thinks I said it..." and titled his memoir.

Re: Google Cloud Networking Incident Postmortem

#163

Post by abbeyj » Fri, Jun 07, 2019, 3:27 PM UTC

dnautics wrote: post #74

carlsborg wrote: post #37

I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1." Answer, and the root cause summarized: Maintenance started in a physical location, and then "... the automation software created a list of jobs to des…

> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.

This reminded me of https://www.usenix.org/system/files/1311_05-08_mickens.pdf

   I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS.

Re: Google Cloud Networking Incident Postmortem

#164

Post by ddalex » Fri, Jun 07, 2019, 3:37 PM UTC

kbirkeland wrote: post #158

oldcreek12 wrote: post #109

Earlier quoted context omitted.

This is inconceivable ... they don't have an OOB management network?

A completely OOB management network is an amazingly high cost when you have presence all over the world. I don't think anybody has gone to the length to double up on dark fiber and OTN gear just for management traffic.

Hmm... With 5G each blade in the rack could get its own modem and sim card for OOB management.

Re: Google Cloud Networking Incident Postmortem

#165

Post by elliekelly » Fri, Jun 07, 2019, 3:44 PM UTC

ljoshua wrote: post #2

Having only ever seen one major outage event in person (at a financial institution that hadn't yet come up with an incident response plan; cue three days of madness), I would love to be a fly on the wall at Google or other well-established engineering orgs when something like this goes down. I'd love to see the red binders come down off the shelf, people organize into incident response groups, and watch as a root cau…

Tangentially related, you might find the documentary "Out of the Clear Blue Sky" interesting. It's about bond trading firm Cantor Fitzgerald (headquartered on the top floors of the World Trade Center) in the days after 9/11. Not even the best plan could have helped them open for business in two days after having lost just about everything. Over the last decade or so we've put a lot of emphasis on documentation when it comes to incident response but the movie is really a testament to how leadership and execution are so much more important.

Re: Google Cloud Networking Incident Postmortem

#166

Post by theflyinghorse » Fri, Jun 07, 2019, 3:56 PM UTC

abbeyj wrote: post #163

dnautics wrote: post #74

Earlier quoted context omitted.

> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.

This reminded me of https://www.usenix.org/system/files/1311_05-08_mickens.pdf I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS.

What an amazing read!

Re: Google Cloud Networking Incident Postmortem

#167

Post by log_n » Fri, Jun 07, 2019, 3:57 PM UTC

mslot wrote: post #149

carlsborg wrote: post #37

I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1." Answer, and the root cause summarized: Maintenance started in a physical location, and then "... the automation software created a list of jobs to des…

At some point people realized servers are prone to failure. They then started deploying their system redundantly to multiple servers in the data center (AZ) to increase availability. This helped, but created consistency issues. To fix this people started building multi-server software systems, creating dependencies across servers that weakened overall availability. At some point people realized multi-server systems w…

Humanity's colonization of the stars will eventually be spurred on by the desire improve the resiliency of SaaS.

Re: Google Cloud Networking Incident Postmortem

#168

Post by dbcurtis » Fri, Jun 07, 2019, 3:58 PM UTC

throwaway_ac wrote: post #97

opportune wrote: post #73

Earlier quoted context omitted.

Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP. Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possib…

I have mixed feelings about the finger pointing/public embarrassment thing. Usually the SRE is matured enough cause they have to be, however the individual teams might not be the same when it comes to reacting/handling the Incident report/postmortem. On a slightly different note, "low-level team to have at least one engineer on call at any given time" - this line itself is so true and at the same time it has so many…

At one Large Org where I worked, the Pager Bearer was paid 25% time for all the time they were on the pager, and standard overtime rates (including weekend/holiday multipliers) from the time the pager went off until they cleared the problem and walked out the plant door, or logged out if the problem was diagnosed/fixed remotely.

25% time for carrying the pager was to compensate for: 1) Requirement to be able to get to the plant in 30 minutes. Fresh snow? Too bad, no skiing for you this weekend. 2) You must be sober and work-ready when the pager goes off. At a party? Great, but I hope you like cranberry juice.

As the customer who signed the time cards for the pager duty, I thought that was not only fair, but it also drove home to me as a manager that the cost was real and was coming out of my budget, not some general IT budget that someone else took the hit for. This is one case where "You want coverage for your service? Give me a charge code for the overtime." was not just senseless bureaucratic friction, it led to healthier, business-driven, decisions.

Re: Google Cloud Networking Incident Postmortem

#169

Post by pishpash » Fri, Jun 07, 2019, 3:59 PM UTC

all_blue_chucks wrote: post #93

exwiki wrote: post #4

Why don't they refund every paid customer who was impacted? Why do they rely on the customer to self report the issue for a refund? For example GCS had 96% packet loss in us-west. So doesn't it make sense to refund every customer who had any API call to a GCS bucket on us-west during the outage?

Why do retailers use mail-in rebates instead of just lowering the price?

That's for price discrimination. This is more like the usual case of don't pay if you don't have to, though in Google's case it could well be that they don't care.

Re: Google Cloud Networking Incident Postmortem

#170

Post by chipperyman573 » Fri, Jun 07, 2019, 4:13 PM UTC

oceliker wrote: post #70

exwiki wrote: post #4

Why don't they refund every paid customer who was impacted? Why do they rely on the customer to self report the issue for a refund? For example GCS had 96% packet loss in us-west. So doesn't it make sense to refund every customer who had any API call to a GCS bucket on us-west during the outage?

Not directly GCS-related, but there was a big Youtube TV outage during the World Cup of last year (I think it was during semi-finals?). Google did apologize, but they only offered a free week of Youtube TV, which they implemented by charging me a week later than usual. I didn't feel compensated at all (it was a pretty important game that I missed!)

What would you prefer they did instead?