Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Normalization of Deviance (2015)

danluu.com

Post reply on HN
Print view View original

21–30 of 228 posts

Re: Normalization of Deviance (2015)

#21

Post by chaps »

aliqot wrote: post #18
chaps wrote: post #4

These stories ring so, so true. Once worked at a company whose infrastructure issues were so deep and festering that after fighting a fire, my boss told me, "If you go to the press about this, the client will sue us and everyone who works here will lose their jobs."

That's what I never understood about this story; did you guys have any suspicion it would dump radiation into the patient all at once, or was this like a concurrency bug

We weren't able to reliably install security daemons on a client's machine because the entire automation system didn't account for autoscaling. The issues were raised well before I joined and the project head legitimately didn't understand it as a problem that needed solving. The hosts were for a presidential candidate's webserver, and they noticed the webservers were missing security daemons days before the election.

Re: Normalization of Deviance (2015)

#22

Post by aliqot »

chaps wrote: post #21
aliqot wrote: post #18

Earlier quoted context omitted.

That's what I never understood about this story; did you guys have any suspicion it would dump radiation into the patient all at once, or was this like a concurrency bug

We weren't able to reliably install security daemons on a client's machine because the entire automation system didn't account for autoscaling. The issues were raised well before I joined and the project head legitimately didn't understand it as a problem that needed solving. The hosts were for a presidential candidate's webserver, and they noticed the webservers were missing security daemons days before the election…

jeez thats a rough spot to be in. did you stick around to fix it or just get the hell out of dodge after that?

Re: Normalization of Deviance (2015)

#23

Post by kurthr »

There's been a lot of work on reliability of complex systems and how they operate. What has been found is that it is almost always necessary to have failure (degraded operation) modes that prevent system failure, and the more complex and more hazardous failure is the more modes develop.

In these systems it is found that they are almost always operating (or transitioning between) failure modes. Often multiple operational failure modes are simultaneous. It becomes very important to test the system in each of it's failure modes and their combinations to maintain high up time.

https://how.complexsystems.fail/ is an example, but there are many.

Human work, development, and maintenance is itself a system that interacts with these critical systems. Frankly, failure to fail causes failure (thus chaos monkey). The mythical man month is almost a sub category of these failures as are HR hiring processes and other BS. Being too successful and not having competition (or similarly sclerotic competition) can be as much of a hazard as "move fast, break things".

Re: Normalization of Deviance (2015)

#25

Post by xnorswap »

AlbertCory wrote: post #13

This guy needs to organize & format his writing better, since he does have really interesting things to say.

If you don't have a reader view in your browser, just paste this into your global CSS:

    p 
    {
      line-height: 1.7;
      max-width: 60em;
      font-size: 1.2em;
      margin-left: 5em;
    }
It pretty much fixes the default readability which is essentially zero on this site otherwise.

Re: Normalization of Deviance (2015)

#26

Post by chaps »

aliqot wrote: post #22
chaps wrote: post #21

Earlier quoted context omitted.

We weren't able to reliably install security daemons on a client's machine because the entire automation system didn't account for autoscaling. The issues were raised well before I joined and the project head legitimately didn't understand it as a problem that needed solving. The hosts were for a presidential candidate's webserver, and they noticed the webservers were missing security daemons days before the election…

jeez thats a rough spot to be in. did you stick around to fix it or just get the hell out of dodge after that?

I did what I could with a handful of selenium scripts, then hit a road block because we didn't have ssh access to a chunk of the autoscaling hosts. Gave up after that, told the customer rep to tell them we can't do it, and gave my two week notice about a month later.

Re: Normalization of Deviance (2015)

#27

Post by kerblang »

Great stuff - I think this goes in the "required reading" list.

The tech industry tends to revolve around "I'm a super-rational robotic genius" thinking that can't accept the existence of its own irrational tendencies, to the point that it becomes ridiculous.

Re: Normalization of Deviance (2015)

#28

Post by AnIdiotOnTheNet »

chaps wrote: post #15
AnIdiotOnTheNet wrote: post #12

Earlier quoted context omitted.

To be fair, you really shouldn't. You know nothing of the constraints that people are operating under, or the political or cultural landscape you're dealing with, so you just come off like a preachy academic.

See the other comment I made: I was explicitly asked to speak up.

If that's the case, then yeah fair enough. If someone doesn't want your opinion they shouldn't ask for it.

Re: Normalization of Deviance (2015)

#29

Post by dec0dedab0de »

chaps wrote: post #14
giobox wrote: post #10

Earlier quoted context omitted.

There's often a wise tradeoff between criticizing systems you've just seen after being at the company 5 minutes and actually spending some time at the company to learn the historical context of why the thing you think is insane/shit is insane/shit before telling everyone who built it how insane/shit it is. People generally don't wake up in the morning and go into work motivated to make insane/shit things - context, t…

You (in fashion of the article) missed the point I made: I was explicitly asked to speak up as the new employee and when I did I was told to stop speaking up. When I brought it up in the meeting I gave my two week notice, he admitted to saying that and apologized.

That's some bullshit. you did the right thing. When a new employee joins and notices that something sucks the answer should be something like these:

We know, but haven't had time to fix it, maybe we'll assign that to you when you're caught up.

We didn't think of it that way, good catch, lets go into detail later.

Yeah, but doing it this way makes this other thing easier, we'll show you that when you're ready.

or even: I don't know, my brain is fried with this project, can you ask again in a few months?

Re: Normalization of Deviance (2015)

#30

Post by chaps »

a4isms wrote: post #19
giobox wrote: post #10

Earlier quoted context omitted.

There's often a wise tradeoff between criticizing systems you've just seen after being at the company 5 minutes and actually spending some time at the company to learn the historical context of why the thing you think is insane/shit is insane/shit before telling everyone who built it how insane/shit it is. People generally don't wake up in the morning and go into work motivated to make insane/shit things - context, t…

"Making good decisions, therefore, requires understanding past decisions. Without knowing how things came to be, it’s easy to make things worse." — https://thoughtbot.com/blog/chestertons-fence

I was once told this after raising the issue that "hey maybe an API that responds with root ssh passwords is a bad idea, and our clients are going to be pissed once they find out." And.. I was right.

So often, citing Chesterton's fence is significantly more naive than what it attempts to criticize.

Post reply on HN