Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Count to ten when a plane goes down

johncbeck.tumblr.com

Post reply on HN
Print view View original

251–260 of 284 posts

Re: Count to ten when a plane goes down

#251

Post by justizin »

It's a sign of poor management that someone has to be fired when something goes wrong, outages are learning situations for all involved, and it is widely held that the person who took the action that caused an outage is not responsible, but that all involved are responsible.

See John Allspaw's Swiss Cheese Theory : http://www.kitchensoap.com/2012/02/10/each-necessary-but-onl... .

[ Edit: I guess it's not Allspaw's model, but he applies it to systems engineering rather well - http://en.wikipedia.org/wiki/Swiss_cheese_model ]

"Accidents emerge from a confluence of conditions and occurrences that are usually associated with the pursuit of success, but in this combination—each necessary but only jointly sufficient—able to trigger failure instead."

The person who pushed the button is not at fault, the manager is not at fault, the guy who designed the button is not at fault - all are jointly responsible.

Blaming the intern does, however, reflect extremely poorly on Itoh and everyone else in the chain of command. A superior who demands retribution for a simple mistake that happened to cause him or her pain is basically worthless.

But, I forget, we're talking about Ronald Reagan.

Re: Count to ten when a plane goes down

#252

Post by woodchuck64 »

nathanb wrote: post #235
noisy_boy wrote: post #207

Firstly, firing the intern doesn't make sense - it was a mistake waiting to happen and he just happened to do it at the wrong time. Secondly, the punishment meted out should be: 1. Proportional to the degree of carelessness (in this case not that much since he accidentally hit a wrong key adjacent to the right one, didn't mow down anybody while driving drunk) 2. Inversely proportional to the likelihood of the error (…

I think you're wrong to ignore the consequences of the actions as an input to the punishment. A small amount of unintentional carelessness that causes huge damage could still be punished. One could argue that a certain degree of mindfulness in critical situations is a job requirement, and casually making a careless -- though unintended -- mistake demonstrates a lack of mindfulness indicating that the person is not pr…

> I think you're wrong to ignore the consequences of the actions as an input to the punishment.

This equivocates on "consequences" of actions, though. It's obvious that the consequences of hitting F7 before the incident were understood by all responsible to be low enough that any intern could be expected to make the right decision. After the incident, the consequences of hitting F7 were sharply increased such that no future intern would ever be allowed to make that decision. But then you can't make an argument that assumes "consequences" were the same at both points in time.

We make this fallacy all the time probably because we're designed by evolution to reassess the morality of an action based on consequences. It works as a social heuristic for shaming or rewarding people but it makes no rational sense that the morality of an action should retroactively change based on future consequences. You can see similar behavior in our rewarding athletes for profound genetic advantages, or punishing criminals for profound genetic deficits. The consequences somehow redeem or condemn, and they should do neither.

Re: Count to ten when a plane goes down

#253

Post by lcedp »

kstenerud wrote: post #215
marbletiles wrote: post #18

Earlier quoted context omitted.

The world where you know you that a) you have an incredibly powerful key with no safeguards at your fingertips and b) you might be in a breaking news situation and nonetheless you go for the key right next to the dangerous one carelessly enough that you miss? Think about it: Unix is equally as "insane". If you're the guy on the console who meant to clean out some crap dir and accidentally typoed "rm -rf /" and then c…

The worst unix disaster I ever saw happened to one of my co-workers. He was working on a client machine, logged in as root because he needed to compile and install some complicated software. As he was working, he did an ls -l /bin and copy-pasted it to a text editor so he could make sure everything was installed correctly. Unfortunately, after returning to his console, he accidentally hit paste. Most of /bin was actu…

That's fascinating.

To prevent this from happening with me, I've added the following line to my `.Xdefaults`

    URxvt.perl-ext: confirm-paste
(I'm using `rxvt-unicode`)

http://i.imgur.com/joHRXaH.png

Re: Count to ten when a plane goes down

#254

Post by ufmace »

mikegreco wrote: post #4

The author states they felt it was appropriate when they were fired. In what world would it be appropriate to get fired for a single, simple, incredibly easy to make mistake? Doubly insane when there were exactly zero safeguards in place to prevent the mistake from being made.

On the other hand, they're hiring people to do these things, not computer programs. Shouldn't we expect them to notice that when doing routine-thing-x, it's awfully easy to accidentally do catastrophically-dangerous-thing-y, and thus it would be a very good idea to be extremely slow and deliberate when doing routine-thing-x?

I have to routinely create and drop databases on my local system. Our production databases, which I also have to connect to, contain hundreds, maybe thousands, of person-months of work. I realized that it would be a good idea, before issuing DROP DATABASE commands, to deliberately stop and double-check what server I'm connected to. Luckily, I haven't screwed that one up yet.

Re: Count to ten when a plane goes down

#255

Post by lotsofmangos »

Loughla wrote: post #198
lamontcg wrote: post #94

Earlier quoted context omitted.

And then conspiracy theorists latch onto this kind of shit, but believe that it must be malicious silliness... Why didn't Reagan respond immediately? Well, he was waiting to hear from Chancellor Gorkon that the KAL flight had been successfully beamed aboard and was en route to Pluto, of course... Clearly they'd have their shit together better so it couldn't have been a 23-year old rebooting all the computers accident…

Conspiracy theorists are just 20th and 21st century prophets, really. They search for meaning in an all too often meaningless world. It's comforting to think that people can control the direction of every choice in the world, and that someone is at the helm. It's uncomfortable to think about the daily series of random, unconnected decisions that drive the direction of our species.

I'm not sure that it is more comforting to think that there is someone at the helm, as much as anyone who aspires to be considered to be at the helm has to keep pushing that story, so it gets repeated more often and with better special effects than the story about there not being anyone at the helm.

Actually being in control of stuff is very difficult, but convincing people that you are in control of stuff is pretty easy as we are all suckers for narrative. The main ways to disrupt a power narrative is to spread other narratives or for a situation to occur that upsets the existing narrative, so getting people to make up new ones. This explains why totalitarian governments can collapse so quickly, which wouldn't be possible if the people running them were actually in control of anything.

Re: Count to ten when a plane goes down

#256

Post by marbletiles »

woodchuck64 wrote: post #252
nathanb wrote: post #235

Earlier quoted context omitted.

I think you're wrong to ignore the consequences of the actions as an input to the punishment. A small amount of unintentional carelessness that causes huge damage could still be punished. One could argue that a certain degree of mindfulness in critical situations is a job requirement, and casually making a careless -- though unintended -- mistake demonstrates a lack of mindfulness indicating that the person is not pr…

> I think you're wrong to ignore the consequences of the actions as an input to the punishment. This equivocates on "consequences" of actions, though. It's obvious that the consequences of hitting F7 before the incident were understood by all responsible to be low enough that any intern could be expected to make the right decision. After the incident, the consequences of hitting F7 were sharply increased such that no…

No, the consequences were the same before and after the incident: a total system reboot. The varying factor here was temporal: it was usually a low-risk action when the office was empty, a high-risk one when the office was full.

The negligence on the intern's part was to make decisions and act without regard for risk as if he was in the low-risk window despite the evidence he was actually in the high-risk one (all the already-active PCs).

It makes perfect sense that the punishment should reflect inappropriate regard being given for known consequences. That's what negligence is.

Re: Count to ten when a plane goes down

#257

Post by [deleted] »

nerfhammer wrote: post #144
mikegreco wrote: post #4

The author states they felt it was appropriate when they were fired. In what world would it be appropriate to get fired for a single, simple, incredibly easy to make mistake? Doubly insane when there were exactly zero safeguards in place to prevent the mistake from being made.

He publicly embarrassed USG, POTUS and a major US ally. SK is going to call up the state dept and demand an explanation. Someone has to be fired. This isn't some startup in California where everyone just plays it cool. The termination of his boss and his boss's boss and his boss's boss's boss all the way up were probably considered as well.

[deleted]

Re: Count to ten when a plane goes down

#258

Post by Spooky23 »

billmalarky wrote: post #231
emn13 wrote: post #180

Earlier quoted context omitted.

I expect the HR management back then was more enlightened than it is now - the quick-to-pounce media and politically instigated witch hunts (terrorism, save-the-kids, etc.) have ensured that cover-your-ass is more and more a necessity.

That sounds like rose colored glasses. The cold war had its fair share of witch hunts. Granted this was the 80's not the 50's but lets not forget McCarthyism was based around democracy vs communism. "That Korean announcement and the slow response by the US President—both caused by delayed real information—caused decades of conspiracy theories."

HR was called "Personnel" in those days. It was different.

The reality here is that an intern doesn't have civil service protection, so it is quick and easy to dispose of them. Going after the supervisor may take longer, so if rapid action is needed, they'll fire the first person who serves "at the pleasure of" the executive.

Re: Count to ten when a plane goes down

#259

Post by zeidrich »

noisy_boy wrote: post #207

Firstly, firing the intern doesn't make sense - it was a mistake waiting to happen and he just happened to do it at the wrong time. Secondly, the punishment meted out should be: 1. Proportional to the degree of carelessness (in this case not that much since he accidentally hit a wrong key adjacent to the right one, didn't mow down anybody while driving drunk) 2. Inversely proportional to the likelihood of the error (…

Punishment is not a good way to correct behavior.

Your number 2 is particularly wrong. For punishment to work in affecting behavior, you can't punish for a very unlikely event, especially accidental.

Punishment changes behavior by making people anxious and afraid of the punishment. If you punish something that's very unlikely, it does no good. It's like if pushing ctrl-F restarted the stations, and the guy has never pushed ctrl-F before, and never been punished for it. It's very unlikely that he would ever push ctrl-F. But he happens to trip getting up and accidentally hit ctrl-F while he's catching his balance. Does that warrant a heavier punishment? It's more unlikely, certainly, but what would the punishment change about his behavior?

Punishment works because you are afraid of it. It works because you want to avoid it. But accidents don't happen because of defiance or a rational decision making process.

If you were punished moderately and frequently for a common mistake like hitting F7, it could correct behavior because you would be more vigilant when hitting F6. Having it be proportional to the degree of carelessness in terms of correcting behavior is not important. If someone is more careless, they will get more frequent punishment. If the punishment is too strong, it will just make people fearful instead of correcting the behavior.

Firing someone who consistently makes mistakes is a corrective action, not punitive.

Punishment is generally more of a cultural thing and less of a means of correcting an issue. Punishment is expected, so it's delivered. In western culture we have a particular need to find someone responsible and punish them. Rarely though do you feel "I don't want to get punished, so I am going to do this right." but it's not uncommon to think "I don't want to get punished so I'll avoid this altogether."

Corrective behavior is better when it's not punitive. Look at the design of the software, correct that problem. Look at the systems that allowed this to happen, correct them. Work with the staff and find out why this could happen, help them correct it. If people are punished for writing the software poorly, they're just going to cover up the flaws that they find instead of bringing them to light to correct them. If staff are punished for making mistakes, they're going to hide them instead of seeing if they can fix them.

Punishment is often just a game to abdicate responsibility. "Oh, it wasn't my fault. It was his fault. The proof that it is his fault is that he got punished for it. I've done my part to solve this problem."

Especially in complex environments like corporations and government, I think that the last thing you should do is look for a person to blame. Instead of looking for the person responsible for implementing the design, or the person who approved it. Look at why it was implemented, how it was approved. Instead of pinning it on an individual, pin it on a system.

I think you should only look at an individual if they are committing malfeasance for the purpose of benefiting themselves outside of the system. If the person approved the design because they weren't aware of the potential risk, then find out why. If they approved it because there was supposed to be another safeguard to stop it from accidentally happening, find out why that wasn't there. If they approved it because they gave the contract to their friend who wasn't the best decision, and overlooked issues for a cut, then go ahead and blame them.

If there's a problem with the person, say the designer was just irreconcilably bad, then remove him. If it's a problem with training, then train him. If it was something he did as a greenhorn in the past, and now he's much better, then for God's sake don't punish him for a mistake he made years ago when he was put into a project that was more important than the skills he was hired with, unless he grossly lied about his skills.

Re: Count to ten when a plane goes down

#260

Post by krylon »

graylights wrote: post #46
baddox wrote: post #36

Earlier quoted context omitted.

But typing "rm -rf /" is significantly harder to do accidentally than typing F7 instead of F6.

Not really, a lot of novice unix users are of the habit of removing files with -rf switch. I cringe everytime I see it. The command "rm -rf ~/blue/" is just a single space key from being equivalent to "rm -rf /" with "rm -rf ~/blue /"

Hehe, I did something like that once, except that I typed "rm -rf ~ /blue/". There was no /blue/, but I managed to wipe out my home directory, and I did not have a backup. :-| It was on my personal machine, so at least I did not delete anybody else's files, but I still got burned hard enough to learn a valuable lesson.
Post reply on HN