Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

To Stop Cheating, Nuclear Officers Ditch the Grades

npr.org

Post reply on HN
Print view View original

11–20 of 61 posts

Re: To Stop Cheating, Nuclear Officers Ditch the Grades

#11

Post by TeMPOraL »

Ntrails wrote: post #10
hliyan wrote: post #4

In other words, you get what you measure. This is one of the reasons I discourage measuring team performance in terms of bug counts or lines-of-code. The former lead to what some call "issue-tennis" and informal fixes, and the latter lead to code bloat. After years of trying, I've finally concluded that "key performance indicators" attached to individuals or teams (as opposed to artifacts and periods) usually leads t…

If the KPIs were, say, something to be minimised - does this still hold? For example, I cannot see a way to game "Lower system downtime is better", or "lower load time is better". Or am I fundamentally misunderstanding what a KPI is?

Sure you can.

- lower system downtime = don't do maintenance or do it in a half-assed way

- decrease system load time = make everything possible "optional post-load components", thus reducing load time by factor of, say, 10, and increasing time to combat readiness by a factor of 5.

People can pervert any metric they're measured by. Just look at what companies do to squeeze that last little dollar of profit out of their customers.

Re: To Stop Cheating, Nuclear Officers Ditch the Grades

#12

Post by arethuza »

jedberg wrote: post #5

So War Games was right then... We really do need to replace the officers with computers.

The Soviets did have an automated system that could launch their missiles automatically:

http://en.wikipedia.org/wiki/Dead_Hand_%28nuclear_war%29

It may still be operational, but not normally turned on - for obvious reasons....

At the other end of the spectrum the UK has no automated control preventing unauthorized launches (no Permissive Action Locks) and we check to see whether Radio 4 is still broadcast to detect whether civilisation has ended and hand written letters to instruct the captains of our Tridents subs what to do in that event.

Re: To Stop Cheating, Nuclear Officers Ditch the Grades

#13

Post by markgraydk »

TeMPOraL wrote: post #11
Ntrails wrote: post #10

Earlier quoted context omitted.

If the KPIs were, say, something to be minimised - does this still hold? For example, I cannot see a way to game "Lower system downtime is better", or "lower load time is better". Or am I fundamentally misunderstanding what a KPI is?

Sure you can. - lower system downtime = don't do maintenance or do it in a half-assed way - decrease system load time = make everything possible "optional post-load components", thus reducing load time by factor of, say, 10, and increasing time to combat readiness by a factor of 5. People can pervert any metric they're measured by. Just look at what companies do to squeeze that last little dollar of profit out of the…

In that case you should have more than just one KPI to combat any perverse incentives. Further, any decision making based on the KPIs should take into account thorough analysis.

It's far from perfect but making things quantifiable does have a lot of benefits.

Re: To Stop Cheating, Nuclear Officers Ditch the Grades

#14

Post by serf »

arethuza wrote: post #12
jedberg wrote: post #5

So War Games was right then... We really do need to replace the officers with computers.

The Soviets did have an automated system that could launch their missiles automatically: http://en.wikipedia.org/wiki/Dead_Hand_%28nuclear_war%29 It may still be operational, but not normally turned on - for obvious reasons.... At the other end of the spectrum the UK has no automated control preventing unauthorized launches (no Permissive Action Locks) and we check to see whether Radio 4 is still broadcast to detect…

The American Project Pluto/SLAM system was meant to be autonomous and self-launching too. The Western MAD system.

Boy am I glad that project never got off the ground. A MAD-gap/race would have been terrifying.

A plane that can circle the Earth 4 and a half times is pretty cool, but at the cost of irradiating the whole planet? Maybe not so much.

Re: To Stop Cheating, Nuclear Officers Ditch the Grades

#15

Post by tomjen3 »

Ntrails wrote: post #10
hliyan wrote: post #4

In other words, you get what you measure. This is one of the reasons I discourage measuring team performance in terms of bug counts or lines-of-code. The former lead to what some call "issue-tennis" and informal fixes, and the latter lead to code bloat. After years of trying, I've finally concluded that "key performance indicators" attached to individuals or teams (as opposed to artifacts and periods) usually leads t…

If the KPIs were, say, something to be minimised - does this still hold? For example, I cannot see a way to game "Lower system downtime is better", or "lower load time is better". Or am I fundamentally misunderstanding what a KPI is?

If I was punished for downtime I wouldn't have upgraded OpenSSL after heartbleed.

I wouldn't put new features online.

I general I would do little that might cause the system to be unstable, even if it was a actually a good thing.

Re: To Stop Cheating, Nuclear Officers Ditch the Grades

#16

Post by watwut »

markgraydk wrote: post #13
TeMPOraL wrote: post #11

Earlier quoted context omitted.

Sure you can. - lower system downtime = don't do maintenance or do it in a half-assed way - decrease system load time = make everything possible "optional post-load components", thus reducing load time by factor of, say, 10, and increasing time to combat readiness by a factor of 5. People can pervert any metric they're measured by. Just look at what companies do to squeeze that last little dollar of profit out of the…

In that case you should have more than just one KPI to combat any perverse incentives. Further, any decision making based on the KPIs should take into account thorough analysis. It's far from perfect but making things quantifiable does have a lot of benefits.

The trouble is, the decisions based on KPI do not take these things into account. They are usually made up in the hierarchy by people entirely detached from what is done day to day. They usually do not know the harm done and wonder where the company is loosing money.

Your combination of multiple KPI amounts to somewhat more complicated KPI. There will be perverse incentives unless you get perfectly right. And that is possible only if the work, including the long term consequences of decisions, are easily quantifiable - which is not the case of most non-trivial jobs.

Re: To Stop Cheating, Nuclear Officers Ditch the Grades

#17

Post by jacquesm »

TeMPOraL wrote: post #11
Ntrails wrote: post #10

Earlier quoted context omitted.

If the KPIs were, say, something to be minimised - does this still hold? For example, I cannot see a way to game "Lower system downtime is better", or "lower load time is better". Or am I fundamentally misunderstanding what a KPI is?

Sure you can. - lower system downtime = don't do maintenance or do it in a half-assed way - decrease system load time = make everything possible "optional post-load components", thus reducing load time by factor of, say, 10, and increasing time to combat readiness by a factor of 5. People can pervert any metric they're measured by. Just look at what companies do to squeeze that last little dollar of profit out of the…

One company I recently worked with had 0 scheduled system down time. They also had about 500 manyears worth of technical debt and had to jump through increasingly complex hoops in order to maintain the 0 down time policy.

Short term that looks pretty good though, longer term it is a guaranteed to fail strategy. When and if it catches up with you you're deep trouble.

Re: To Stop Cheating, Nuclear Officers Ditch the Grades

#18

Post by jib »

lobotryas wrote: post #8
WalterBright wrote: post #7

Earlier quoted context omitted.

I learned that 30 years ago. It seems to be one of those things that every generation needs to learn the hard way. A reason why an effective team needs a few older guys who've been around the block a few times :-)

True. Unfortunately you then get a new VP who still believes in KPIs (or needs to demonstrate KPIs to his boss because that's the only thing upper management "gets") and then you're back to square one :(

It is fine to have KPIs. When KPIs go out of control it is a sign of culture issues, or whatever you want to call it.

KPIs are just that. Indicators. Temperature is an indicator of good climate. If someone's answer to "it is cold in here" is to put a lighter under the thermometer then the problem isn't the thermometer, right? It is the people who are confusing indicators from goals. And that is a culture thing.

If that is your climate though you can still recover from it (I know we are doing way better than before at least:)).

First steps are to embrace ambiguity and uncertainty in evaluations. The culture twist (when I have seen it) comes from "Noone recognises my work" -> "Lets define KPIs for your work" -> "I am not evaluated fairly, look at how much work I do compared to guys that are evaluated better" -> "Lets define guidelines for evaluations based on our KPIs" -> "Everyone (except me) is cheating to score high on the KPIs to get better evaluations" -> "We need better KPIs that can't be cheated on" -> "We are doing a whole amount of stupid stuff because people worry about KPIs".

I think the first two steps are fine. There is direct business value in being able to compare how you are doing compared to the previous month, or some other department or whatever.

The problem comes in the third step - once the KPIs determine how someone is evaluated, then you are in a downwards spiral, because now you are saying that how you get there isn't as important as getting there. To get out of that you need to embrace the ambiguity that comes with letting evaluations happen, and maybe be "unfair".

Basically - uncertainty in evaluations is better than defining evaluations on KPIs. Solve the uncertainly by more discussions and reasoning, not by more rules.

Re: To Stop Cheating, Nuclear Officers Ditch the Grades

#19

Post by markgraydk »

watwut wrote: post #16
markgraydk wrote: post #13

Earlier quoted context omitted.

In that case you should have more than just one KPI to combat any perverse incentives. Further, any decision making based on the KPIs should take into account thorough analysis. It's far from perfect but making things quantifiable does have a lot of benefits.

The trouble is, the decisions based on KPI do not take these things into account. They are usually made up in the hierarchy by people entirely detached from what is done day to day. They usually do not know the harm done and wonder where the company is loosing money. Your combination of multiple KPI amounts to somewhat more complicated KPI. There will be perverse incentives unless you get perfectly right. And that is…

I agree that KPIs are misused quite a lot but that does change the fact that they can provide valuable information and setup incentives to follow certain goals.

I think it is rather simplistic to say that having more than one KPI can be reduced to "a more complicate KPI". Sure, it won't solve all problems with badly setup KPIs but to allow for detailed incentive structures you need detailed KPIs.

My major point is still that decision making based on KPIs should not be made solely on what the figures (or any red/yellow/green flag next to it) show. It requires analysis - which is something that I've always seen done.

The good thing about KPIs is that it can be a great communication tool and open discusions about why a trend has changed or similar things.

Re: To Stop Cheating, Nuclear Officers Ditch the Grades

#20

Post by TeMPOraL »

markgraydk wrote: post #19
watwut wrote: post #16

Earlier quoted context omitted.

The trouble is, the decisions based on KPI do not take these things into account. They are usually made up in the hierarchy by people entirely detached from what is done day to day. They usually do not know the harm done and wonder where the company is loosing money. Your combination of multiple KPI amounts to somewhat more complicated KPI. There will be perverse incentives unless you get perfectly right. And that is…

I agree that KPIs are misused quite a lot but that does change the fact that they can provide valuable information and setup incentives to follow certain goals. I think it is rather simplistic to say that having more than one KPI can be reduced to "a more complicate KPI". Sure, it won't solve all problems with badly setup KPIs but to allow for detailed incentive structures you need detailed KPIs. My major point is st…

> I agree that KPIs are misused quite a lot but that does change the fact that they can provide valuable information and setup incentives to follow certain goals.

The problem is that KPIs are misused like, 99.9% of the times, starting from elementary school. It's rare to see someone using such scores as (one of many) evaluation input and not an output. It's good that you put effort into doing real analysis; sadly, not many people do that.

I think that KPIs, or grades, let you as a manager/evaluator be lazy. You don't have to think, to investigate why the scores are bad, to talk with poor performers. Probably you'll be even discouraged from doing so by your superiors / the whole system - you are not allowed to completely disregard the scores at your discretion.

Post reply on HN