Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Cloud Networking Incident Postmortem

status.cloud.google.com

Post reply on HN
Print view View original

111–120 of 190 posts

Re: Google Cloud Networking Incident Postmortem

#111

Post by lucb1e »

mehrdadn wrote: post #5
exwiki wrote: post #4

Why don't they refund every paid customer who was impacted? Why do they rely on the customer to self report the issue for a refund? For example GCS had 96% packet loss in us-west. So doesn't it make sense to refund every customer who had any API call to a GCS bucket on us-west during the outage?

Probably because it seems to be in the SLA that the customer must notify Google? https://cloud.google.com/storage/sla > "[Customer Must Request Financial Credit] In order to receive any of the Financial Credits described above, Customer must notify Google technical support within thirty days from the time Customer becomes eligible to receive a Financial Credit. Failure to comply with this requirement will forfeit Cus…

So the answer to why it is this way, is because they wrote it down this way..? I think the real question was why this decision was made, not whether they announced this.

Re: Google Cloud Networking Incident Postmortem

#112

Post by donavanm »

scotchio wrote: post #21

Is there a resource that compares all the cloud platform’s reliability? Like a rank and chart of downtime and trends. Just curious how they compare

There are a handful of companies that will try and sell you this. However Id say anything thats simple enough to be expressed as a chart or 1 page summary is not actually useful. Interesting outages have variable breadth, scope, and severity. Its usually some methods or a subset of customers that are impacted. Thats really hard to communicate as a straight percentage. You need to map it back to your particular workload and dependencies. And the meaningful result is how your particular application or customer experience would be affected.

Source: Im a principal at AWS, historically focused on infrastructure and availability/operations, have been oncall for 20 years, and do some internal incident management as my job.

Re: Google Cloud Networking Incident Postmortem

#113

Post by Moru »

virgilp wrote: post #100
throwaway_ac wrote: post #97

Earlier quoted context omitted.

I have mixed feelings about the finger pointing/public embarrassment thing. Usually the SRE is matured enough cause they have to be, however the individual teams might not be the same when it comes to reacting/handling the Incident report/postmortem. On a slightly different note, "low-level team to have at least one engineer on call at any given time" - this line itself is so true and at the same time it has so many…

It was paid or time off where I worked before. It's just being established where I work now, but what's discussed is 2x regular pay for working outside your work hours due to an incident. Doesn't seem "slavery" to me.

In the places I have been working at (lots of different types of jobs), overtime used to be 2x pay or 2x off. None of them were IT-related though.

Re: Google Cloud Networking Incident Postmortem

#114

Post by smueller1234 »

V-eHGsd_ wrote: post #43
iandanforth wrote: post #39

I want a "24" style realtime movie of this event. Call it "Outage" and follow engineers across the globe struggling to bring back critical infrastructure.

it's pretty boring. real life computers aren't at all like hackers or csi:cyber. except for the skateboards, all real sysadmins ride skateboards.

No, not at Google. Lore has it in our office that it used to be hacked high speed segways. At least until somebody decided to build a ramp to jump with and the fiery chariot was reduced to rubble. The driver lived.

Re: Google Cloud Networking Incident Postmortem

#115

Post by the-rc »

panthaaaa wrote: post #83

The defense in depth philosophy means we have robust backup plans for handling failure of such tools, but use of these backup plans ( including engineers travelling to secure facilities designed to withstand the most catastrophic failures , and a reduction in priority of less critical network traffic classes to reduce congestion) added to the time spent debugging. Does that mean engineers travelling to a (off-site) b…

It's either that or special rooms at an office that have a different/redundant setup. Remember that this happened on a Sunday, so most engineers dealing with the incident were home or elsewhere, at least initially.

Re: Google Cloud Networking Incident Postmortem

#116

Post by achiang »

ljoshua wrote: post #2

Having only ever seen one major outage event in person (at a financial institution that hadn't yet come up with an incident response plan; cue three days of madness), I would love to be a fly on the wall at Google or other well-established engineering orgs when something like this goes down. I'd love to see the red binders come down off the shelf, people organize into incident response groups, and watch as a root cau…

Google SRE doesn't have magical incident response beans that we hoard from the rest of the world. What makes Google SRE institutionally strong is that we have senior executive support to execute on all the best practices described in the book:

https://landing.google.com/sre/sre-book/toc/index.html

At my last job, I bought a copy of this book, but we only had the organizational bandwidth to do a few of the things mentioned. At Google, we do all of them.

The incident on Sunday basically played out as described in chapters 13 and 14. There is always the fog of war that exists during an incident, so no, it wasn't always people calmly typing into terminals, but having good structure in place keeps the madness manageable.

Disclosure: I work in Google NetInfra SRE, and while my department was/is heavily involved in this incident, I personally was not.

Also, we're [always] hiring:

https://careers.google.com/jobs/results/?company=Google&comp...

Re: Google Cloud Networking Incident Postmortem

#117

Post by marksomnian »

mentat wrote: post #17
foota wrote: post #12

Earlier quoted context omitted.

I would say this was covered by "Other Google Cloud services which depend on Google's US network were also impacted" it sounds to me like the list of regions was specifically speaking towards loss of connectivity to instances.

It says there wasn't regional congestion, running a function in europe-west2 going to europe-west2 regional bucket is dependent on US network? That would be surprising.

Probably various billing services that need to talk to the mothership in us-east1.

Re: Google Cloud Networking Incident Postmortem

#119

Post by baddox »

all_blue_chucks wrote: post #93
exwiki wrote: post #4

Why don't they refund every paid customer who was impacted? Why do they rely on the customer to self report the issue for a refund? For example GCS had 96% packet loss in us-west. So doesn't it make sense to refund every customer who had any API call to a GCS bucket on us-west during the outage?

Why do retailers use mail-in rebates instead of just lowering the price?

It’s probably more than just “some buyers will forget or not bother to redeem the rebate.” There are other reasons, like price discrimination.

Re: Google Cloud Networking Incident Postmortem

#120

Post by Cthulhu_ »

raverbashing wrote: post #103
dnautics wrote: post #74

Earlier quoted context omitted.

> Debugging the problem was significantly hampered by failure of tools competing over use of the now-congested network. Man that's got to suck.

This happens way more often than you think A very simple example, you do something stupid on a remote machine (either high network usage or CPU usage) over SSH then you can't undo it because SSH becomes unresponsive

Is there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?
Post reply on HN