Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Teaching a new way to prevent outages at Google

sre.google

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Teaching a new way to prevent outages at Google

#31

Post by hansmayer »

mianos wrote: post #5

This is peak corporate drivel—bloated storytelling, buzzwords everywhere, and a desperate attempt to make an old idea sound revolutionary. The article spends paragraphs on some childhood radio repair story before awkwardly linking it to STPA, a safety analysis method that’s been around for decades. Google didn’t invent it, but they act like adapting it for software is a major breakthrough. Most of the piece is just f…

The point about basic engineering concepts is spot on. But I wonder how much it has to do with the creeping in of superficially educated "tech" people across technology sector. Not to downplay the value of self-learning (am a bit of autodidact myself), but the amount of people who switch into the mythical "tech" who have never heard of a differential equation is worrying. Hence companies unfortunately really seem to need to explain concepts like feedback loop to people who only ever heard of it in the context of performance review. The article itself is a word salad though, the start reads like a SEO-optimised cooking blog ;)

Re: Teaching a new way to prevent outages at Google

#33

Post by perching_aix »

cynicalsecurity wrote: post #27
decimalenough wrote: post #19

Earlier quoted context omitted.

If you're referring to the time they nuked an Australian retirement fund's VMware setup, no, that was basically a billing screwup. An operator left a field blank, the system assumed that meant a 1-year expiry, and dutifully deleted it after 1 year was up. https://cloud.google.com/blog/products/infrastructure/detail...

The most unbelievable thing about that case was that Google actually deleted data instead of keeping then forever and use for ads.

Username checks out.

Re: Teaching a new way to prevent outages at Google

#34

Post by sepositus »

praptak wrote: post #28
dataflow wrote: post #18

Earlier quoted context omitted.

By "had a hand in causing" do you mean "they should have prevented it", or do you just mean "they were involved in the causation"? Because sometimes you're forced to do things you know are wrong, because that's what other people are making you do, and in that case you still "have a hand" in causing.

Something in between. Like "pushed to implement a feature without the safety measures". When outages started to happen implemented Outage Prevention Program, i.e. implemented the safety measures that should have been implemented from the start. Subsequent data collection demonstrated X% outage frequency drop clearly demonstrating readiness for promotion, data driven.

Exactly this.

Re: Teaching a new way to prevent outages at Google

#36

Post by [deleted] »

snorkel wrote: post #30

In other words STPA is a design review framework for finding some less obvious failure modes. FMEA is more popular but relies on making a list of all of the knowable failure modes in a system, but the failure modes you haven’t thought of don’t make it on the list. STPA helps fill in some of those gaps of failure modes you haven’t thought of.

[deleted]

Re: Teaching a new way to prevent outages at Google

#37

Post by AStonesThrow »

sepositus wrote: post #8
mianos wrote: post #5

This is peak corporate drivel—bloated storytelling, buzzwords everywhere, and a desperate attempt to make an old idea sound revolutionary. The article spends paragraphs on some childhood radio repair story before awkwardly linking it to STPA, a safety analysis method that’s been around for decades. Google didn’t invent it, but they act like adapting it for software is a major breakthrough. Most of the piece is just f…

I'm not sure if things have changed over the past five years, but this is exactly the stuff you'd throw in a promotion packet or maybe in a performance (perf) review to hit that mythical "superb" rating. The breaking point for me (and why I left after almost a decade) was when people started getting high ratings for fixing things they had an original hand in causing. Honestly, the comfiest job in the world if you're…

It's not easy or popular to link Dilbert these days, but there's a classic cartoon of the PHB announcing their bug bounty program for dev employees, and one of the fellows exclaims that he's going to “code his way to a minivan”!

Re: Teaching a new way to prevent outages at Google

#38

Post by twalla »

SlightlyLeftPad wrote: post #12
irjustin wrote: post #7

I don't understand and I really really want to. This seems so cool at a scale that I can't fathom. Tell me specifically how it's done at google with regards to a specific service, at least enough information to understand what's going on. Make it concrete. Like "B lacks feedback from C", why is this bad? You've told me absolutely nothing and it makes me angry.

This has really always been the case with Google philosophy docs. They tend to be very abstract and academic. The biggest danger is taking everything at face value and structuring your work or organization the same exact way based solely on these documents. The reality is, the vast majority of companies are not Google and will never encounter Google’s problems. That’s not where the value is though.

The other thing to consider is a lot of the time these docs are sort of guidelines or wishlists for the way things ought to be - meanwhile an outside observer will assume these are the way things actually are.

Re: Teaching a new way to prevent outages at Google

#39

Post by tekla »

hansmayer wrote: post #31
mianos wrote: post #5

This is peak corporate drivel—bloated storytelling, buzzwords everywhere, and a desperate attempt to make an old idea sound revolutionary. The article spends paragraphs on some childhood radio repair story before awkwardly linking it to STPA, a safety analysis method that’s been around for decades. Google didn’t invent it, but they act like adapting it for software is a major breakthrough. Most of the piece is just f…

The point about basic engineering concepts is spot on. But I wonder how much it has to do with the creeping in of superficially educated "tech" people across technology sector. Not to downplay the value of self-learning (am a bit of autodidact myself), but the amount of people who switch into the mythical "tech" who have never heard of a differential equation is worrying. Hence companies unfortunately really seem to…

Woah, hold up, why does anyone need to know math?

Re: Teaching a new way to prevent outages at Google

#40

Post by nthingtohide »

decimalenough wrote: post #19
hinkley wrote: post #9

> In one particular case at Google, a software controller–acting on bad feedback from another software system–determined that it should issue an unsafe control action. It scheduled this action to happen after 30 days. Even though there were indicators that this unsafe action was going to occur, no software engineers–humans–were actually monitoring the indicators. So, after 30 days, the unsafe control action occurred,…

If you're referring to the time they nuked an Australian retirement fund's VMware setup, no, that was basically a billing screwup. An operator left a field blank, the system assumed that meant a 1-year expiry, and dutifully deleted it after 1 year was up. https://cloud.google.com/blog/products/infrastructure/detail...

All mega deletes should be authorised. A human person should type in the word "delete" and then only the action should take place. Not doing this is like the decision is taken by VOID created by complex interacting systems.
Post reply on HN