Google Cloud Networking Incident Postmortem • Hacker News Board

Re: Google Cloud Networking Incident Postmortem

#91

Post by Jamesanon » Fri, Jun 07, 2019, 5:16 AM UTC

deathhand wrote: post #45

My burning question is what is a "relatively rare maintenance event type"?

Total speculation and just my interpretation, of course.

What it means to me is that initially some unusually poor decisions were made that triggered an unfortunate and unavoidable events. Very rare is a damage control statement. There is a subtle tone of concern and feeling of blame trough that entire postmortem. This will be buried but if it was investigated thoroughly I wouldn’t be surprised of some serious consequences.

Total speculation. I do not work for google.

Re: Google Cloud Networking Incident Postmortem

#92

Post by hansflying » Fri, Jun 07, 2019, 5:19 AM UTC

eeg3 wrote: post #25

scotchio wrote: post #21

Is there a resource that compares all the cloud platform’s reliability? Like a rank and chart of downtime and trends. Just curious how they compare

There is this from May from Network World: https://www.networkworld.com/article/3394341/when-it-comes-t... GCP was basically even with AWS, and Microsoft was ~6x their downtime according to that article.

Thank you for linking this paid article. GCP on pair with AWS what a joke...

Re: Google Cloud Networking Incident Postmortem

#93

Post by all_blue_chucks » Fri, Jun 07, 2019, 5:21 AM UTC

exwiki wrote: post #4

Why don't they refund every paid customer who was impacted? Why do they rely on the customer to self report the issue for a refund? For example GCS had 96% packet loss in us-west. So doesn't it make sense to refund every customer who had any API call to a GCS bucket on us-west during the outage?

Why do retailers use mail-in rebates instead of just lowering the price?

Re: Google Cloud Networking Incident Postmortem

#94

Post by rhizome » Fri, Jun 07, 2019, 5:23 AM UTC

ernsheong wrote: post #55

exwiki wrote: post #4

Why don't they refund every paid customer who was impacted? Why do they rely on the customer to self report the issue for a refund? For example GCS had 96% packet loss in us-west. So doesn't it make sense to refund every customer who had any API call to a GCS bucket on us-west during the outage?

Customer having to request for refund has been documented in their SLA, e.g. https://cloud.google.com/compute/sla Having said that, if Google wants to delight customers, they should give a free tier bonus to all customers for a certain period, but such a thing cannot be fair to everyone.

>Having said that, if Google wants to delight customers

It'd never happen, "delight" is an Apple principle.

Re: Google Cloud Networking Incident Postmortem

#95

Post by karlding » Fri, Jun 07, 2019, 5:30 AM UTC

londons_explore wrote: post #82

dnautics wrote: post #74

Earlier quoted context omitted.

> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.

It isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of the network control plane.

I don't work as an SRE, but isn't that covered by providing engineers physical access to secure facilities in the absolute worst case?

The article states:

> The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging.

Re: Google Cloud Networking Incident Postmortem

#96

Post by [deleted] » Fri, Jun 07, 2019, 5:39 AM UTC

tobych wrote: post #81

alasdair_ wrote: post #66

Earlier quoted context omitted.

Our own version of Netflix's "Chaos Money" is named "Eris" for precisely the reason mentioned in your first footnote.

Chaos Monkey: https://netflix.github.io/chaosmonkey/

[deleted]

Re: Google Cloud Networking Incident Postmortem

#97

Post by throwaway_ac » Fri, Jun 07, 2019, 5:42 AM UTC

opportune wrote: post #73

roganartu wrote: post #68

Earlier quoted context omitted.

I used to be an SRE at Atlassian in Sydney on a team that regularly dealt with high-severity incidents, and I was an incident manager for probably 5-10 high severity Jira cloud incidents during my tenure too, so perhaps I can give some insight. I left because the SRE org in general at the time was too reactionary, but their incident response process was quite mature (perhaps by necessity). The first thing I'll say is…

Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP. Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possib…

I have mixed feelings about the finger pointing/public embarrassment thing. Usually the SRE is matured enough cause they have to be, however the individual teams might not be the same when it comes to reacting/handling the Incident report/postmortem.

On a slightly different note, "low-level team to have at least one engineer on call at any given time" - this line itself is so true and at the same time it has so many things wrong. Not sure what the best way to put the modern day slavery into words given that I have yet not seen any large org giving day off's for the low-level team engineer just cause they were on call.

Re: Google Cloud Networking Incident Postmortem

#98

Post by tetha » Fri, Jun 07, 2019, 5:49 AM UTC

ethbro wrote: post #88

opportune wrote: post #73

Earlier quoted context omitted.

Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP. Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possib…

I've only been tangentially pulled into high severity incidents, but the thing that most impressed me was the quiet . As mentioned in this thread, it's a lot like listening to air traffic comm chatter. People say what they know , and only what they know, and clearly identify anything they're unsure about. Informative and clear communication matters more than brilliance. Most of the traffic is async task identificatio…

Someone at our place recently commented that the ops team during an incident strongly feels like NASA mission control in critical moments[1]. I wanted to protest, but that's surprisingly accurate.

> And if anyone is screaming or gets emotional, they should not be in that room.

If someone starts yelling around in my incident war room for no reason, they get thrown out. I'm a calm and quiet person, but bugging around during a major incident is one of the few things that make me mad.

1: https://youtu.be/Y0yOTanzx-s?t=3059

Re: Google Cloud Networking Incident Postmortem

#99

Post by lytfyre » Fri, Jun 07, 2019, 5:50 AM UTC

ljoshua wrote: post #2

Having only ever seen one major outage event in person (at a financial institution that hadn't yet come up with an incident response plan; cue three days of madness), I would love to be a fly on the wall at Google or other well-established engineering orgs when something like this goes down. I'd love to see the red binders come down off the shelf, people organize into incident response groups, and watch as a root cau…

You might be interested in https://response.pagerduty.com/, PagerDuty's major incident response process documentation - a good starting point for that red binder.

Having been in the ringmasters seat for major incidents ranging from "relatively routine" to "it's all on fire", and had a ringside seat for a cloud provider outage of comparable magnitude to this one - it still fascinates me how creative solutions can get dreamed up under high pressure, and how effective someone to keep the response calm and _feel like it's in control_ is.

Re: Google Cloud Networking Incident Postmortem

#100

Post by virgilp » Fri, Jun 07, 2019, 6:04 AM UTC

throwaway_ac wrote: post #97

opportune wrote: post #73

Earlier quoted context omitted.

Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP. Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possib…

I have mixed feelings about the finger pointing/public embarrassment thing. Usually the SRE is matured enough cause they have to be, however the individual teams might not be the same when it comes to reacting/handling the Incident report/postmortem. On a slightly different note, "low-level team to have at least one engineer on call at any given time" - this line itself is so true and at the same time it has so many…

It was paid or time off where I worked before. It's just being established where I work now, but what's discussed is 2x regular pay for working outside your work hours due to an incident. Doesn't seem "slavery" to me.