Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teaching a new way to prevent outages at Google

sre.google

Post reply on HN
Print view View original

21–30 of 44 posts

Re: Teaching a new way to prevent outages at Google

#21

Post by bbkane »

SlightlyLeftPad wrote: post #12
irjustin wrote: post #7

I don't understand and I really really want to. This seems so cool at a scale that I can't fathom. Tell me specifically how it's done at google with regards to a specific service, at least enough information to understand what's going on. Make it concrete. Like "B lacks feedback from C", why is this bad? You've told me absolutely nothing and it makes me angry.

This has really always been the case with Google philosophy docs. They tend to be very abstract and academic. The biggest danger is taking everything at face value and structuring your work or organization the same exact way based solely on these documents. The reality is, the vast majority of companies are not Google and will never encounter Google’s problems. That’s not where the value is though.

Maybe less of a philosophy doc, but I found the Google SRE workbook to have plenty of helpful concrete examples

Re: Teaching a new way to prevent outages at Google

#22

Post by z3t4 »

SlightlyLeftPad wrote: post #13
sepositus wrote: post #8

Earlier quoted context omitted.

I'm not sure if things have changed over the past five years, but this is exactly the stuff you'd throw in a promotion packet or maybe in a performance (perf) review to hit that mythical "superb" rating. The breaking point for me (and why I left after almost a decade) was when people started getting high ratings for fixing things they had an original hand in causing. Honestly, the comfiest job in the world if you're…

What I’ve been seeing from Google’s products lately suggests that these are the only ones still there. It’s a house of cards built with professional bullshitters. Google’s culture has entered or is already deep within the bullshit era.

It will happen in all companies that has a monopoly status. If they start to struggle they will just increase the rent.

Re: Teaching a new way to prevent outages at Google

#23

Post by [deleted] »

mimikatz wrote: post #15

Thanks to all the people here pointing out how bloated, overly broad and useless this is. I went to read it thinking I would pick up something applicable and it was written in such a overwrought humanless style that I gave up learning nothing and thought the problem was me. I am glad to learn I am not alone.

[deleted]

Re: Teaching a new way to prevent outages at Google

#24

Post by [deleted] »

SlightlyLeftPad wrote: post #12
irjustin wrote: post #7

I don't understand and I really really want to. This seems so cool at a scale that I can't fathom. Tell me specifically how it's done at google with regards to a specific service, at least enough information to understand what's going on. Make it concrete. Like "B lacks feedback from C", why is this bad? You've told me absolutely nothing and it makes me angry.

This has really always been the case with Google philosophy docs. They tend to be very abstract and academic. The biggest danger is taking everything at face value and structuring your work or organization the same exact way based solely on these documents. The reality is, the vast majority of companies are not Google and will never encounter Google’s problems. That’s not where the value is though.

[deleted]

Re: Teaching a new way to prevent outages at Google

#25

Post by [deleted] »

SlightlyLeftPad wrote: post #12
irjustin wrote: post #7

I don't understand and I really really want to. This seems so cool at a scale that I can't fathom. Tell me specifically how it's done at google with regards to a specific service, at least enough information to understand what's going on. Make it concrete. Like "B lacks feedback from C", why is this bad? You've told me absolutely nothing and it makes me angry.

This has really always been the case with Google philosophy docs. They tend to be very abstract and academic. The biggest danger is taking everything at face value and structuring your work or organization the same exact way based solely on these documents. The reality is, the vast majority of companies are not Google and will never encounter Google’s problems. That’s not where the value is though.

[deleted]

Re: Teaching a new way to prevent outages at Google

#27

Post by cynicalsecurity »

decimalenough wrote: post #19
hinkley wrote: post #9

> In one particular case at Google, a software controller–acting on bad feedback from another software system–determined that it should issue an unsafe control action. It scheduled this action to happen after 30 days. Even though there were indicators that this unsafe action was going to occur, no software engineers–humans–were actually monitoring the indicators. So, after 30 days, the unsafe control action occurred,…

If you're referring to the time they nuked an Australian retirement fund's VMware setup, no, that was basically a billing screwup. An operator left a field blank, the system assumed that meant a 1-year expiry, and dutifully deleted it after 1 year was up. https://cloud.google.com/blog/products/infrastructure/detail...

The most unbelievable thing about that case was that Google actually deleted data instead of keeping then forever and use for ads.

Re: Teaching a new way to prevent outages at Google

#28

Post by praptak »

dataflow wrote: post #18
sepositus wrote: post #8

Earlier quoted context omitted.

I'm not sure if things have changed over the past five years, but this is exactly the stuff you'd throw in a promotion packet or maybe in a performance (perf) review to hit that mythical "superb" rating. The breaking point for me (and why I left after almost a decade) was when people started getting high ratings for fixing things they had an original hand in causing. Honestly, the comfiest job in the world if you're…

By "had a hand in causing" do you mean "they should have prevented it", or do you just mean "they were involved in the causation"? Because sometimes you're forced to do things you know are wrong, because that's what other people are making you do, and in that case you still "have a hand" in causing.

Something in between. Like "pushed to implement a feature without the safety measures". When outages started to happen implemented Outage Prevention Program, i.e. implemented the safety measures that should have been implemented from the start.

Subsequent data collection demonstrated X% outage frequency drop clearly demonstrating readiness for promotion, data driven.

Re: Teaching a new way to prevent outages at Google

#29

Post by dooglius »

> After working with the system experts to build this control structure, we immediately noticed missing feedback from controller C to controller B–in other words, controller B did not have enough information to support the decisions it needed to make

There is a feedback loop through D? And why does the same issue not apply to the missing directed edge from B to D?

EDIT: I figured it out on a reread, the vertical up/down orientation matters for whether an edge represents control vs feedback, so B is merely not controlling D, which is fine. But if B is only controlling C as a way to get through to D (which is what I would have guessed, absent other information), what's the issue with that?

Re: Teaching a new way to prevent outages at Google

#30

Post by snorkel »

In other words STPA is a design review framework for finding some less obvious failure modes. FMEA is more popular but relies on making a list of all of the knowable failure modes in a system, but the failure modes you haven’t thought of don’t make it on the list. STPA helps fill in some of those gaps of failure modes you haven’t thought of.
Post reply on HN