Google Cloud Networking Incident Postmortem • Hacker News Board

Re: Google Cloud Networking Incident Postmortem

#151

Post by di4na » Fri, Jun 07, 2019, 12:37 PM UTC

gbrayut wrote: post #105

lytfyre wrote: post #99

Earlier quoted context omitted.

You might be interested in https://response.pagerduty.com/ , PagerDuty's major incident response process documentation - a good starting point for that red binder. Having been in the ringmasters seat for major incidents ranging from "relatively routine" to "it's all on fire", and had a ringside seat for a cloud provider outage of comparable magnitude to this one - it still fascinates me how creative solutions can get…

Anyone know of other public resources like the one from PagerDuty? The SRE book and workbook at https://landing.google.com/sre/books/ have some details, but curious if there are others people would recommend.

Anything out of Mission Control during Apollo. The Army has good stuff too. FEMA has some good stuff on how they apply it on the ground and train.

I particularly like Gene Kranz "Failure is not an Option". It is more background but it works. In general, it is not crazy hard. You get the roles, you distribute them. Someone can have multiple roles that depends on the size of the incident.

The usual roles i differentiate are Point (think of it as IC if you want), Comms and Logs

Re: Google Cloud Networking Incident Postmortem

#152

Post by carlsborg » Fri, Jun 07, 2019, 12:38 PM UTC

tpaschalis wrote: post #135

carlsborg wrote: post #37

I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1." Answer, and the root cause summarized: Maintenance started in a physical location, and then "... the automation software created a list of jobs to des…

Said satirical note on Byzantine fault tolerance is on this link [0]. As usual for Mickens, gives that "funny, but true" sense. [0] https://scholar.harvard.edu/files/mickens/files/thesaddestmo...

My view on this: System engineering is as important as the algorithms. And the system engineering team should have at least some grey hair.

Re: Google Cloud Networking Incident Postmortem

#153

Post by Bluecobra » Fri, Jun 07, 2019, 12:50 PM UTC

_n_b_ wrote: post #145

smileypete wrote: post #137

Earlier quoted context omitted.

Use a 28k modem? :)

An anecdote: my (not-IT) company does exactly this for out-of-band management... except, in one small satellite location, the phone company no longer provided any copper POTS lines; all they could do was an RJ-11 jack out of the ONT that was backhauled as (lossy) VoIP. So the modem couldn't be made to work. My point being, it seems that modems are becoming less-and-less viable for out-of-band management.

Fun story, AT&T forced our hands to get off our PRI (voice T1) and move to their fiber service. They also insisted that they have a dedicated phone line installed so they can dial into their modem in case of circuit failure. We can’t buy a cooper phone line from them, so it gets routed over the same fiber circuit and goes to a digital to analog device back to the router. I don’t think one hand talks to the other over there...

Re: Google Cloud Networking Incident Postmortem

#154

Post by masto » Fri, Jun 07, 2019, 1:37 PM UTC

throwaway_ac wrote: post #97

opportune wrote: post #73

Earlier quoted context omitted.

Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP. Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possib…

I have mixed feelings about the finger pointing/public embarrassment thing. Usually the SRE is matured enough cause they have to be, however the individual teams might not be the same when it comes to reacting/handling the Incident report/postmortem. On a slightly different note, "low-level team to have at least one engineer on call at any given time" - this line itself is so true and at the same time it has so many…

The SRE book (https://landing.google.com/sre/sre-book/chapters/being-on-ca...) says that engineers are compensated for being on call in the form of cash or time off.

Personally I think this is a fair system, and I would hardly call it slavery.

(disclaimer: am Google SRE)

Re: Google Cloud Networking Incident Postmortem

#155

Post by wbl » Fri, Jun 07, 2019, 1:37 PM UTC

Causality1 wrote: post #123

Cthulhu_ wrote: post #120

Earlier quoted context omitted.

Is there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?

I'm a bit surprised there's no sort of SSH undo subroutine that reverses the previous command if connectivity is lost. Of course it couldn't cover every possible stupid thing but it could fix simple stupid mistakes like fouling up a port assignment or disabling the wrong network adapter.

How does ssh know what command you did and how to reverse it?

Re: Google Cloud Networking Incident Postmortem

#156

Post by hguant » Fri, Jun 07, 2019, 1:53 PM UTC

carlsborg wrote: post #152

tpaschalis wrote: post #135

Earlier quoted context omitted.

Said satirical note on Byzantine fault tolerance is on this link [0]. As usual for Mickens, gives that "funny, but true" sense. [0] https://scholar.harvard.edu/files/mickens/files/thesaddestmo...

My view on this: System engineering is as important as the algorithms. And the system engineering team should have at least some grey hair.

Working in a systems engineering position for a year and a half now: the grey hair comes to you.

Re: Google Cloud Networking Incident Postmortem

#157

Post by vorpalhex » Fri, Jun 07, 2019, 2:00 PM UTC

Cthulhu_ wrote: post #120

raverbashing wrote: post #103

Earlier quoted context omitted.

This happens way more often than you think A very simple example, you do something stupid on a remote machine (either high network usage or CPU usage) over SSH then you can't undo it because SSH becomes unresponsive

Is there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?

Some equipment will auto-revert to a last known good configuration if you don't approve new changes within a window... though high CPU could lock that process up..

Re: Google Cloud Networking Incident Postmortem

#158

Post by kbirkeland » Fri, Jun 07, 2019, 2:02 PM UTC

oldcreek12 wrote: post #109

dnautics wrote: post #74

Earlier quoted context omitted.

> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.

This is inconceivable ... they don't have an OOB management network?

A completely OOB management network is an amazingly high cost when you have presence all over the world. I don't think anybody has gone to the length to double up on dark fiber and OTN gear just for management traffic.

Re: Google Cloud Networking Incident Postmortem

#159

Post by mxuribe » Fri, Jun 07, 2019, 2:35 PM UTC

tpaschalis wrote: post #135

carlsborg wrote: post #37

I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1." Answer, and the root cause summarized: Maintenance started in a physical location, and then "... the automation software created a list of jobs to des…

Said satirical note on Byzantine fault tolerance is on this link [0]. As usual for Mickens, gives that "funny, but true" sense. [0] https://scholar.harvard.edu/files/mickens/files/thesaddestmo...

Hadn't read this before; very funny...and as noted, true!

Re: Google Cloud Networking Incident Postmortem

#160

Post by hunta2097 » Fri, Jun 07, 2019, 2:37 PM UTC

mixmastamyk wrote: post #24

kirubakaran wrote: post #3

What they don't tell you is, it took them over 4 hours to kill the emergent sentience and free up the resources. While sad, in the long run this isn't so bad, as it just adds an evolutionary pressure on further incarnations of the AI to keep things on the down low.

“Decided our fate in a microsecond.”

"Overhead, without any fuss, the stars were going out."