Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Networking Incident Postmortem

status.cloud.google.com

Post reply on HN
Print view View original

81–90 of 190 posts

Re: Google Cloud Networking Incident Postmortem

#81

Post by tobych »

alasdair_ wrote: post #66
tazjin wrote: post #13

Earlier quoted context omitted.

It's interesting to see it go down. There's some chaos involved, but from my perspective it's the constructive[0] kind. If you're interested in how these sorts of incidents are managed, check out the SRE Book[1] - it has a chapter or two on this and many other related topics. Disclosure: I work in Google Cloud, but not SRE. [0]: https://principiadiscordia.com/book/70.php [1]: https://landing.google.com/sre/books/

Our own version of Netflix's "Chaos Money" is named "Eris" for precisely the reason mentioned in your first footnote.

Chaos Monkey: https://netflix.github.io/chaosmonkey/

Re: Google Cloud Networking Incident Postmortem

#82

Post by londons_explore »

dnautics wrote: post #74
carlsborg wrote: post #37

I was curious to know how cascading failures in one region effected other regions. Impact was " ...increased latency, intermittent errors, and connectivity loss to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1, and southamerica-east1." Answer, and the root cause summarized: Maintenance started in a physical location, and then "... the automation software created a list of jobs to des…

> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.

It isn't a worst-case though. They should have had the capability to resolve this issue with no network connectivity, which would be the worst case failure of the network control plane.

Re: Google Cloud Networking Incident Postmortem

#83

Post by panthaaaa »

The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans (including engineers travelling to secure facilities designed to withstand the most catastrophic failures, and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging.

Does that mean engineers travelling to a (off-site) bunker?

Re: Google Cloud Networking Incident Postmortem

#84

Post by rorykoehler »

roganartu wrote: post #68
ljoshua wrote: post #2

Having only ever seen one major outage event in person (at a financial institution that hadn't yet come up with an incident response plan; cue three days of madness), I would love to be a fly on the wall at Google or other well-established engineering orgs when something like this goes down. I'd love to see the red binders come down off the shelf, people organize into incident response groups, and watch as a root cau…

I used to be an SRE at Atlassian in Sydney on a team that regularly dealt with high-severity incidents, and I was an incident manager for probably 5-10 high severity Jira cloud incidents during my tenure too, so perhaps I can give some insight. I left because the SRE org in general at the time was too reactionary, but their incident response process was quite mature (perhaps by necessity). The first thing I'll say is…

> I left because the SRE org in general at the time was too reactionary

It shows in their products (though it's improving)

Re: Google Cloud Networking Incident Postmortem

#85

Post by munchbunny »

thegabez wrote: post #57
V-eHGsd_ wrote: post #43

Earlier quoted context omitted.

it's pretty boring. real life computers aren't at all like hackers or csi:cyber. except for the skateboards, all real sysadmins ride skateboards.

What?! It's the most exciting part of the job. Entire departments coming together, working as a team to problem solve under duress. What's more exciting than that?

Having done it at both big companies and startups... honestly, the startup version is more interesting. Higher stakes, more resourcefulness required, more swearing, and more camaraderie. The incidents I've been a part of in big company contexts have been pretty undramatic - tons of focus on keeping emotions muted, carefully unpeeling the onion, and then carefully sequencing mitigations and repairs.

Re: Google Cloud Networking Incident Postmortem

#86

Post by Wintereise »

fouc wrote: post #78
anonfunction wrote: post #72

The only way to get SLA credits is requesting it. This is very disappointing. SLA CREDITS If you believe your paid application experienced an SLA violation as a result of this incident, please populate the SLA credit request: https://support.google.com/cloud/contact/cloud_platform_sla

That does seem questionable. They should be able to detect who was affected in the first place.

They can. It's a cost minimization thing, a LOT of people don't want to bother with requesting despite being eligible.

This prevents people from pointing the finger at them for not providing SLA credits.

Re: Google Cloud Networking Incident Postmortem

#87

Post by jbigelow76 »

Wintereise wrote: post #86
fouc wrote: post #78

Earlier quoted context omitted.

That does seem questionable. They should be able to detect who was affected in the first place.

They can. It's a cost minimization thing, a LOT of people don't want to bother with requesting despite being eligible. This prevents people from pointing the finger at them for not providing SLA credits.

SLACreditRequestsAAS? Who's with me, all I need is a co-founder and an eight million dollar series A round to last long enough that a cloud provider buys us up before they actually have to pay out a request!

Re: Google Cloud Networking Incident Postmortem

#88

Post by ethbro »

opportune wrote: post #73
roganartu wrote: post #68

Earlier quoted context omitted.

I used to be an SRE at Atlassian in Sydney on a team that regularly dealt with high-severity incidents, and I was an incident manager for probably 5-10 high severity Jira cloud incidents during my tenure too, so perhaps I can give some insight. I left because the SRE org in general at the time was too reactionary, but their incident response process was quite mature (perhaps by necessity). The first thing I'll say is…

Yes this is also how it's done at other large orgs. But one key to a quick response is for every low-level team to have at least one engineer on call at any given time. This makes it so any SRE team can engage with true "owners" of the offending code ASAP. Also during an incident, fingers are never publicly/embarrassingly pointed nor are people blamed. It's all about identifying and fixing the issue as fast as possib…

I've only been tangentially pulled into high severity incidents, but the thing that most impressed me was the quiet.

As mentioned in this thread, it's a lot like listening to air traffic comm chatter.

People say what they know, and only what they know, and clearly identify anything they're unsure about. Informative and clear communication matters more than brilliance.

Most of the traffic is async task identification, dispatch, and then reporting in.

And if anyone is screaming or gets emotional, they should not be in that room.

Re: Google Cloud Networking Incident Postmortem

#89

Post by hansflying »

Google has a huge quality problem and their service is extremely unreliable. Another 3-day-outage in kubernetes:

https://news.ycombinator.com/item?id=18428497

login issues:

https://news.ycombinator.com/item?id=19687029

storage system outage:

https://news.ycombinator.com/item?id=19392452

...

So, basically Google created the most unreliable cloud system in the world.

Re: Google Cloud Networking Incident Postmortem

#90

Post by ethbro »

ehsankia wrote: post #56
V-eHGsd_ wrote: post #43

Earlier quoted context omitted.

it's pretty boring. real life computers aren't at all like hackers or csi:cyber. except for the skateboards, all real sysadmins ride skateboards.

Is it real skateboards or boosted boards (or those one wheeled electric boards?).

I believe parent was probably referencing this: https://m.youtube.com/watch?v=kV_i8AefT8I

But in defense, why be admin if you don't look admin?

Post reply on HN