How Completely Messed Up Practices Become Normal • Hacker News Board

Re: How Completely Messed Up Practices Become Normal

#91

Post by smaddox » Wed, Dec 30, 2015, 6:07 PM UTC

satai wrote: post #68

cubano wrote: post #60

My personal hottake is that, as Einstein proved, everything really is relative. What may seem dysfunctional and WTF from your vantage point may be perfectly logical from someone elses. If you are going thru your work-life with the idea that social relationships and business practices are always going to follow the same strict rules as your software or hardware does, well good luck with that. Normal is as normal does.

Einstein proved nothing like this.

Next you'll tell me that Descartes didn't prove the existence of Zeus!

Re: How Completely Messed Up Practices Become Normal

#92

Post by devonkim » Wed, Dec 30, 2015, 6:15 PM UTC

koko775 wrote: post #74

devonkim wrote: post #69

Earlier quoted context omitted.

Is the implication that 2 9s are really bad, mediocre, or some other negative judgment in this context? I'm familiar with a lot of places that have trouble with even 1 9 of reliability with lots of capital but terrible culture despite employing lots of operational controls and some best practices that have trouble with one 9, so even 2 9s sounds like a dream sometimes and the way that Dan Luu worded this article make…

How does a place survive being down 37 days per year (one nine)? Cripes!

SLA that defines unplanned maintenance very, very conservatively. For example, a place I was at did maintenance for about 8 hours weekly where users wouldn't be able to manage any of their stuff, but there was network access outbound possible from their resources, so it was considered "available." Somehow with hardly anyone actually doing anything with the service, it was a major part of an $800M+ acquisition and mostly for being "enterprise" ready. I will not understand how you can get away with things that demonstrably fail to work beyond the most trivial of canned demo cases and be sold like it's completely done to even more gullible companies with zero technical vetting.

There's a lot of completely inconcistent definitions about what a service is sold as and what is actually delivered, and lawyers half the time only care about regulatory requirements rather than functional ones. Someone will use the ITIL definitions of service and say "it's available!" meaning that it exists in a CMDB or something and another person defines availability as "I can ping it" and doesn't care if there's an HTTP 500 error being thrown repeatedly. But gosh, if something used an insecure MongoDB server that is the reason they immediately cancel a contract (not hyperbole, saw something very similar happen).

Re: How Completely Messed Up Practices Become Normal

#93

Post by dfc » Wed, Dec 30, 2015, 6:17 PM UTC

momzer wrote: post #49

hyperpape wrote: post #30

Earlier quoted context omitted.

It's a little bit awkward. Still, I find it oddly fascinating that you're confused, because I can't figure out what's confusing. There is a company Dan Luu knows about. This company has a reputation for great engineering practices. This company had 2 9s of reliability when Dan last checked. The reason it has 2 9s of reliability is a predictable result of its engineering practices. Although this example is about a spe…

The unspoken implication here is that 99% reliability is considered bad. This may not be clear if coming from a different field where 99% sounds pretty good.

When you say 99% is bad, are you making a normative or positive statement?

Re: How Completely Messed Up Practices Become Normal

#94

Post by taurath » Wed, Dec 30, 2015, 6:18 PM UTC

devonkim wrote: post #45

danso wrote: post #10

I immediately thought of the "5 monkeys and a ladder" psychology study, in which the first monkey attempts to climb the ladder to get a banana and all of the monkeys are sprayed with water. Then one of the monkeys is replaced with a newcomer, who then tries to climb the ladder, but this time, the remaining original monkeys attack it. And so the replacing of the monkeys continues, until none of the monkeys knows what…

This appears related to learned helplessness except on a sociological basis rather than individual basis.

And as a corollary the moment someone breaks an "unbreakable" record in many sports that record can sometimes be frequently broken right after.

Re: How Completely Messed Up Practices Become Normal

#95

Post by Wistar » Wed, Dec 30, 2015, 6:22 PM UTC

jwmerrill wrote: post #8

The paper on "normalization of deviance" that this post links to is also really good. It's written from a medicine perspective, but its observations and conclusions are pretty generalizable. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2821100/

An interesting (to me) recent piece of writing from the aviation world also points a finger at the "normalization of deviance." Written by Ron Rapp, a charter pilot:

http://www.rapp.org/archives/2015/12/normalization-of-devian...

Re: How Completely Messed Up Practices Become Normal

#96

Post by agiamas » Wed, Dec 30, 2015, 6:22 PM UTC

calinet6 wrote: post #88

shalmanese wrote: post #47

Earlier quoted context omitted.

This is exactly a perfect example of WTF worthy behavior being treated as totally normal; that it's encoded that there's "leadership" and "not leadership" as binary job distinctions with distinct responsibilities. I've been in both types of cultures and, by far the healthier culture was the one in which "making the organization better" was treated as everyone's responsibility. I've seen very junior engineers drive qu…

Yep. There's a whole philosophy based on the idea that improvement of the organization is everyone's job. It's called Lean, or more simply, continuous improvement or Kaizen.

yeah there are a gazillion companies that claim to be Lean, Kaizen or what else but guess what, in practice only a small fraction actually are :)

Re: How Completely Messed Up Practices Become Normal

#97

Post by devonkim » Wed, Dec 30, 2015, 6:34 PM UTC

taurath wrote: post #94

devonkim wrote: post #45

Earlier quoted context omitted.

This appears related to learned helplessness except on a sociological basis rather than individual basis.

And as a corollary the moment someone breaks an "unbreakable" record in many sports that record can sometimes be frequently broken right after.

Competitive eating is a great example of this. Never really thought this carried over in general to other forms of sport / competition.

Re: How Completely Messed Up Practices Become Normal

#98

Post by apalmer » Wed, Dec 30, 2015, 6:35 PM UTC

shalmanese wrote: post #47

vinceguidry wrote: post #15

Most of what the author is complaining about boils down to business needs being more important than the needs of the engineering team. To be blunt, they're paying you to do a job, not to make the organization better. That's what they pay the leadership for. You want to be part of the leadership, work your way through the ranks or start your own business. My life got much, much easier once I learned to stop straining…

This is exactly a perfect example of WTF worthy behavior being treated as totally normal; that it's encoded that there's "leadership" and "not leadership" as binary job distinctions with distinct responsibilities. I've been in both types of cultures and, by far the healthier culture was the one in which "making the organization better" was treated as everyone's responsibility. I've seen very junior engineers drive qu…

Leadership leads. This is a tautology. It is unhealthy to pretend otherwise.

Lets take for Leadership decides to incur technical debt to allow a faster entry into market, Development Team Lead decides this is unacceptable, and does not take on the Technical Debt, but instead does things 'The Right Way'. This leads to competitors entering the market first, snagging customers and mindshare, which eventually leads to the company going under.

Was the Development Team 'Right'? Even if lets say he was 'Right', is it his decision to make, should he intentionally sabotage or ignore direction from his superiors ?

Most true WTF processes tend to come from decisions made that are out of the control of the parties that they directly impact.

Re: How Completely Messed Up Practices Become Normal

#99

Post by protonfish » Wed, Dec 30, 2015, 6:36 PM UTC

I think there is another thing to take away from some of the case studies - if you design and implement operating procedures and alarms, do so in a way that is simple, effective, and does not draw an undue amount of time and attention to itself. I have dealt with too many systems that sound the "everything's OK alarm" constantly and procedures that have good intentions but no effort to streamline the gratuitous amount of time and effort needed to be followed.

It is not constructive to blame employees for failure to heed poor alerts and protocols.

Re: How Completely Messed Up Practices Become Normal

#100

Post by hyperpape » Wed, Dec 30, 2015, 6:37 PM UTC

dfc wrote: post #70

hyperpape wrote: post #30

Earlier quoted context omitted.

It's a little bit awkward. Still, I find it oddly fascinating that you're confused, because I can't figure out what's confusing. There is a company Dan Luu knows about. This company has a reputation for great engineering practices. This company had 2 9s of reliability when Dan last checked. The reason it has 2 9s of reliability is a predictable result of its engineering practices. Although this example is about a spe…

So, characteristic X is so common that it does reveal the identity of company Y . Yet the author is surprised that many companies think X is normal?

There are two points I'd make:

1) Maybe yes. If the practices are really bad, it could be surprising that they're widespread and that people don't see a problem.

2) In general, when someone is pointing out something bad, saying "you're surprised?!" is counter-productive. You don't have to be surprised to call something out (I'm not surprised anymore how much our government spies on us, but it seems bad).