Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Networking Incident Postmortem

status.cloud.google.com

Post reply on HN
Print view View original

121–130 of 190 posts

Re: Google Cloud Networking Incident Postmortem

#121

Post by the-rc »

deathhand wrote: post #45

My burning question is what is a "relatively rare maintenance event type"?

My hunch: a more invasive one. Think of turning off all machines in a cluster for major power work or to replace the enclosures themselves. Maintenance on a single machine or rack, instead, happens all the time and requires little more scheduling work than what you do in Kubernetes when you drain a node or a group of nodes. I used to have my share of "fun" at Google sometimes when clusters came back unclean from major maintenance. That usually had no customer-facing impact, because traffic had been routed somewhere else the entire time.

Re: Google Cloud Networking Incident Postmortem

#122

Post by maccam94 »

jbigelow76 wrote: post #87
Wintereise wrote: post #86

Earlier quoted context omitted.

They can. It's a cost minimization thing, a LOT of people don't want to bother with requesting despite being eligible. This prevents people from pointing the finger at them for not providing SLA credits.

SLACreditRequestsAAS? Who's with me, all I need is a co-founder and an eight million dollar series A round to last long enough that a cloud provider buys us up before they actually have to pay out a request!

This exists for Comcast and some other stuff, I'll ask my roommate what the service is called.

Re: Google Cloud Networking Incident Postmortem

#123

Post by Causality1 »

Cthulhu_ wrote: post #120
raverbashing wrote: post #103

Earlier quoted context omitted.

This happens way more often than you think A very simple example, you do something stupid on a remote machine (either high network usage or CPU usage) over SSH then you can't undo it because SSH becomes unresponsive

Is there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?

I'm a bit surprised there's no sort of SSH undo subroutine that reverses the previous command if connectivity is lost. Of course it couldn't cover every possible stupid thing but it could fix simple stupid mistakes like fouling up a port assignment or disabling the wrong network adapter.

Re: Google Cloud Networking Incident Postmortem

#124

Post by ti_ranger »

eeg3 wrote: post #25
scotchio wrote: post #21

Is there a resource that compares all the cloud platform’s reliability? Like a rank and chart of downtime and trends. Just curious how they compare

There is this from May from Network World: https://www.networkworld.com/article/3394341/when-it-comes-t... GCP was basically even with AWS, and Microsoft was ~6x their downtime according to that article.

From the article:

> AWS has the most granular reporting, as it shows every service in every region. If an incident occurs that impacts three services, all three of those services would light up red. If those were unavailable for one hour, AWS would record three hours of downtime.

Was this reflected in their bar graph or not?

Also, GCP has had a number of global events, e.g. the inability to modify any load balancer for >3 hours last year, which AWS has NEVER had (unless you count when AWS was the only cloud with one region).

Re: Google Cloud Networking Incident Postmortem

#125

Post by Spearchucker »

Cthulhu_ wrote: post #120
raverbashing wrote: post #103

Earlier quoted context omitted.

This happens way more often than you think A very simple example, you do something stupid on a remote machine (either high network usage or CPU usage) over SSH then you can't undo it because SSH becomes unresponsive

Is there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?

Recovery-oriented computing. Seperate Bohr bugs (easy to reproduce) from Heisenbugs (difficult to reproduce/change their behaviour once investigated).

Bohr bugs generate an alert and happily meander through normal support channels.

Heisenbugs go through phases -

1. Probation. On continued failure,

2. Restart. If the app or service fails after a restart,

3. Reboot. If the app or service fails after a reboot,

4. Re-image. If the app or service fails after re-imaging,

5. Remove/elimate the node.

Re: Google Cloud Networking Incident Postmortem

#126

Post by topranks »

Causality1 wrote: post #123
Cthulhu_ wrote: post #120

Earlier quoted context omitted.

Is there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?

I'm a bit surprised there's no sort of SSH undo subroutine that reverses the previous command if connectivity is lost. Of course it couldn't cover every possible stupid thing but it could fix simple stupid mistakes like fouling up a port assignment or disabling the wrong network adapter.

A commit / confirm / rollback cycle is how this is typically dealt with in network automation.

Re: Google Cloud Networking Incident Postmortem

#127

Post by wikibob »

KirinDave wrote: post #108
throwaway_ac wrote: post #97

Earlier quoted context omitted.

I have mixed feelings about the finger pointing/public embarrassment thing. Usually the SRE is matured enough cause they have to be, however the individual teams might not be the same when it comes to reacting/handling the Incident report/postmortem. On a slightly different note, "low-level team to have at least one engineer on call at any given time" - this line itself is so true and at the same time it has so many…

Having recently joined an SRE team at Google with a very large oncall component, fwiw I think the policies around oncall are fair and well-thought-out. There is an understanding of how it impacts your time, your energy and your life that is impressive? To be honest, I feel bad for being so macho about oncall at the org I ran and just having the leads take it all upon ourselves.

What are the policies exactly? I’ve heard it’s equal time off for every night you are on call?

Re: Google Cloud Networking Incident Postmortem

#128

Post by pas »

person_of_color wrote: post #102

As a electronics/firmware engineer, is there a dummies resource than covers this concept of a "cloud"?

Besides the completely valid GNU link, the important bits are:

- the cloud is just a bunch of computers, managed by someone. either you (on-premise private cloud) or by someone else as a SaaS

- building, operating, managing, administering, maintaining a cloud is hard (look at the OpenStack project, it's a "success", but very much a non-competitor, because you still need skilled IT labor, there's no real one-size-fits all, so you need to basically maintain your own fork/setup and components - see eg what Rackspace does)

- it's a big security, scalability and stability problem thrown under the bus of economics (multi-tenant environments are hard to price, hard to secure and hard to scale; shared resources like network bandwidth and storage operations-per-sec make no sense to dedicate, because then you need dedicated resources not shared - which is of course just allocated from a bigger shared pool, but then you have to manage the competing allocations)

Re: Google Cloud Networking Incident Postmortem

#130

Post by Moru »

tetha wrote: post #98
ethbro wrote: post #88

Earlier quoted context omitted.

I've only been tangentially pulled into high severity incidents, but the thing that most impressed me was the quiet . As mentioned in this thread, it's a lot like listening to air traffic comm chatter. People say what they know , and only what they know, and clearly identify anything they're unsure about. Informative and clear communication matters more than brilliance. Most of the traffic is async task identificatio…

Someone at our place recently commented that the ops team during an incident strongly feels like NASA mission control in critical moments[1]. I wanted to protest, but that's surprisingly accurate. > And if anyone is screaming or gets emotional, they should not be in that room. If someone starts yelling around in my incident war room for no reason, they get thrown out. I'm a calm and quiet person, but bugging around d…

When the insident is over, does it look like 55:50 in that video? :-)
Post reply on HN