Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why you've never been in a plane crash

asteriskmag.com

Post reply on HN
Print view View original

131–140 of 310 posts

Re: Why you've never been in a plane crash

#131

Post by weezin »

roenxi wrote: post #66
throwaway713 wrote: post #34

This concept of a blameless culture reminds me of one time when I was talking to a SWE at Facebook around 2010. I don’t know whether the story is actually true or just folklore, but apparently someone brought down the whole site on accident once, and it was pretty obvious who did it. Zuckerberg was in the office and walked up to the guy and said something along the lines of “Just so you are aware, it would probably t…

Someone get Zuck an above-average-manager award. But that isn't quite what you want in a blameless culture. The right response looks something like ignoring the engineer, gathering the tech leads and having an extremely detailed walkthrough of exactly what went wrong, how they managed to put an engineer in a position where an expensive outage happened and then they explain why it is never going to happen again. And a…

Personally I feel like the right thing to do is let the engineer closest to the incident lead the response and subsequent action items. If they do well commend them, if they don't take it seriously then it may be time to look for a new job.

Re: Why you've never been in a plane crash

#132

Post by BillyTheKing »

dustedcodes wrote: post #72
throwaway713 wrote: post #34

This concept of a blameless culture reminds me of one time when I was talking to a SWE at Facebook around 2010. I don’t know whether the story is actually true or just folklore, but apparently someone brought down the whole site on accident once, and it was pretty obvious who did it. Zuckerberg was in the office and walked up to the guy and said something along the lines of “Just so you are aware, it would probably t…

Whenever this happens to someone it's always a horrible feeling where one feels very guilty and ashamed no matter what people say to you and unfortunately mistakes like these are almost the bread and butter of any extremely experienced grey beard so it's kind of normal that something like this happens to someone at some point sooner or later. The only people who never make costly mistakes are those who were never tru…

> the cost which often gets quoted with those mistakes is not a real cost

Oh that depends entirely on the industry.. in social media maybe not, in banking and Fintech those can most certainly be real costs. And can tell you - that feels even worse

Re: Why you've never been in a plane crash

#133

Post by InitialLastName »

946789987649 wrote: post #117
roenxi wrote: post #66

Earlier quoted context omitted.

Someone get Zuck an above-average-manager award. But that isn't quite what you want in a blameless culture. The right response looks something like ignoring the engineer, gathering the tech leads and having an extremely detailed walkthrough of exactly what went wrong, how they managed to put an engineer in a position where an expensive outage happened and then they explain why it is never going to happen again. And a…

I would say you need to acknowledge and talk to the engineer. They will be stressed and upset, highlighting there is no blame will ease that.

They'll also be the person most able to identify what went wrong with your processes to allow the failure to occur and think through a mechanism to systematically avoid it happening again.

Also, they're probably the person least likely to make that class of mistake again. If you can keep them, you've added a lot of experiential value to your team.

Re: Why you've never been in a plane crash

#134

Post by rft »

sgjohnson wrote: post #100
mrjin wrote: post #60

Earlier quoted context omitted.

I would assume the pilots of those still had control to some extent?

Total loss of control basically never happens. I’ve been reading and watching a lot of videos on aircraft accidents and near misses, and I haven’t seen a single one where there truly was a total loss of all controls over the aircraft. Even in the case of Japan Airlines Flight 123 ( https://en.m.wikipedia.org/wiki/Japan_Air_Lines_Flight_123 ) the pilots still had some degree of control over the plane after its tail wa…

Also see United Airlines Flight 232 https://en.wikipedia.org/wiki/United_Airlines_Flight_232 which suffered a loss of all hydraulics and thus control surfaces. The crew and a check airmen who was aboard as a passenger managed to somewhat control the plane using engine thrust and managed to steer the plane to an airport before making a crash landing. They saved more than half the people on the plane.

One of the harrowing details is this quote: "The crew contacted United Airlines maintenance personnel via radio, but were told that the possibility of a total loss of hydraulics in a DC-10 was considered so remote that no procedure was established for such an event."

Re: Why you've never been in a plane crash

#135

Post by ta1243 »

Jensson wrote: post #108
dustedcodes wrote: post #72

Earlier quoted context omitted.

Whenever this happens to someone it's always a horrible feeling where one feels very guilty and ashamed no matter what people say to you and unfortunately mistakes like these are almost the bread and butter of any extremely experienced grey beard so it's kind of normal that something like this happens to someone at some point sooner or later. The only people who never make costly mistakes are those who were never tru…

> the cost which often gets quoted with those mistakes is not a real cost It is still money they would have made that now weren't made. It is very important to explain to people how much value is lost during these events so that we also correctly value the work to prevent such events in the future.

If your shop takes $3600 an hour in revenue, but there's a problem with the till which means that people can't pay for 10 seconds, you haven't lost $10 in revenue, you've just shifted revenue from $1/second to $0/second for 10 seconds and $2/second for the next 10 seconds.

Re: Why you've never been in a plane crash

#136

Post by randomdata »

throwaway713 wrote: post #34

This concept of a blameless culture reminds me of one time when I was talking to a SWE at Facebook around 2010. I don’t know whether the story is actually true or just folklore, but apparently someone brought down the whole site on accident once, and it was pretty obvious who did it. Zuckerberg was in the office and walked up to the guy and said something along the lines of “Just so you are aware, it would probably t…

> it would probably take a lifetime or more to recoup the revenue lost during that outage.

Must be hard times at Facebook if a short period of downtime in 2010 still hasn't been recouped.

Re: Why you've never been in a plane crash

#137

Post by furyofantares »

kqr wrote: post #109
Jensson wrote: post #108

Earlier quoted context omitted.

> the cost which often gets quoted with those mistakes is not a real cost It is still money they would have made that now weren't made. It is very important to explain to people how much value is lost during these events so that we also correctly value the work to prevent such events in the future.

You're comparing "reality where accident happened" to "an alternate reality where everything is exactly the same but the accident did not happen" and this is not a sensible comparison. The reality we have produced the accident. You can't have that reality and have it not produce the accident, because it was set up to produce the accident. Proof: it produced the accident. To avoid the accident, you need an alternative…

And to add to that - the number is almost always wrong because people tend to just count the money hose throughput times the downtime. But many of the people who would have spent money on the downtime will do so later. I guess maybe that's not true of advertising revenue? Although I imagine advertisers tend to have some monthly spend.

Re: Why you've never been in a plane crash

#138

Post by ta1243 »

hayst4ck wrote: post #98
latexr wrote: post #68

Earlier quoted context omitted.

> four million dollars That looks like a value inflated by retellings of the popular Thomas Watson quote (see sibling comments).

Four million is definitely in the range of an outage at peak, that's not counting reallocated engineering resources to root cause and fix the problem, the opportunity cost of that fix in lost features, extra work by PR, potential contractual obligations for uptime, outage aftershocks, recruiting implications, customer support implications, etc. If you have a once a year outage, how many employee-hours do you think yo…

By that measure a loss of 1 second would be $7k lost revenue. A 100ms outage would be $700.

Reality is that revenue wouldn't be lost if you had a 100ms outage, it wouldn't even be noticed.

Re: Why you've never been in a plane crash

#139

Post by mikepurvis »

ta1243 wrote: post #135
Jensson wrote: post #108

Earlier quoted context omitted.

> the cost which often gets quoted with those mistakes is not a real cost It is still money they would have made that now weren't made. It is very important to explain to people how much value is lost during these events so that we also correctly value the work to prevent such events in the future.

If your shop takes $3600 an hour in revenue, but there's a problem with the till which means that people can't pay for 10 seconds, you haven't lost $10 in revenue, you've just shifted revenue from $1/second to $0/second for 10 seconds and $2/second for the next 10 seconds.

Yup, the only "real" cost there is a customer who decides not to buy after all, or buys elsewhere instead. But that's pretty unlikely, especially for short outages. And it's even less of an issue for entities with a lot of stickiness like social networks (Facebook, Twitter) or shopping websites with robust loyalty programs (Amazon Prime).

Re: Why you've never been in a plane crash

#140

Post by User23 »

Jensson wrote: post #108
dustedcodes wrote: post #72

Earlier quoted context omitted.

Whenever this happens to someone it's always a horrible feeling where one feels very guilty and ashamed no matter what people say to you and unfortunately mistakes like these are almost the bread and butter of any extremely experienced grey beard so it's kind of normal that something like this happens to someone at some point sooner or later. The only people who never make costly mistakes are those who were never tru…

> the cost which often gets quoted with those mistakes is not a real cost It is still money they would have made that now weren't made. It is very important to explain to people how much value is lost during these events so that we also correctly value the work to prevent such events in the future.

It's also hard to understand because it's largely illusory. If, say, Facebook is down and ad spending ceases for an hour, that money didn't just go up in smoke. It's still in somebody's ad budget, and there's a very good chance they're still going to spend it on ads. Thus, while there will be a temporary slow down in ad spend rate, over the course of the quarter the average may be completely unaffected due to catch up spending to use the budget.
Post reply on HN