Classical example of Goodhart's law: http://en.wikipedia.org/wiki/Goodhart%27s_law
I would think that Campbell’s law is a slightly better fit: https://en.wikipedia.org/wiki/Campbell%27s_law
And a good example is probably Google ranking...
To Stop Cheating, Nuclear Officers Ditch the Grades
21–30 of 61 posts
Re: To Stop Cheating, Nuclear Officers Ditch the Grades
#22So War Games was right then... We really do need to replace the officers with computers.
The Soviets did have an automated system that could launch their missiles automatically: http://en.wikipedia.org/wiki/Dead_Hand_%28nuclear_war%29 It may still be operational, but not normally turned on - for obvious reasons.... At the other end of the spectrum the UK has no automated control preventing unauthorized launches (no Permissive Action Locks) and we check to see whether Radio 4 is still broadcast to detect…
(off-topic, but I find it fascinating)
The UK system sounds back-asswards, but it strikes me as a surprisingly sane set of checks.
Man vs Machine has always been a royal navy doctrine. The US teaches nuclear engineers how to drive a boat. The UK teaches sailors how to run a nuke. Similarly, if the boat has to maintain independent firing capability (the whole point of the deterrent being in submarines is to avoid command&control decapitation), then any access control is a formality. If control ultimately falls to the crew, then don't pretend it doesn't - focus on the men.
Radio 4 sounds anachronistic, but it's a UK-based, govt-controlled transmitter, independent from military systems, which broadcasts every second of the day during routine operation - and because it's still cranked out on longwave (198?khz), can be heard across our waters & much of the north atlantic (the stomping grounds of our deterrent).
While civilian maritime broadcasts would sound like a more sensible failsafe, most of them are either broadcast or controlled from Northwood, so they're not independent of military systems, and are routinely silent between scheduled broadcasts.
What I do find interesting is the actual point of checking for civilian broadcasts - given the loss or failure of command&control communications, the result isn't a railroad to Armageddon - they're expected to implement some common sense. Compare this to the scenes in Crimson Tide, where Gene Hackman prefers blind obedience to the Protocol ..
And "hand-written" is a red-herring. The actual detail here is that only the PM ever knows what went in that letter once it's sealed. They don't have to be hand-written, and probably aren't anymore - but traditionally this was a side effect of not dictating to a secretary/typist.
Re: To Stop Cheating, Nuclear Officers Ditch the Grades
#23In other words, you get what you measure. This is one of the reasons I discourage measuring team performance in terms of bug counts or lines-of-code. The former lead to what some call "issue-tennis" and informal fixes, and the latter lead to code bloat. After years of trying, I've finally concluded that "key performance indicators" attached to individuals or teams (as opposed to artifacts and periods) usually leads t…
So I was in language courses in Cairo for a while. I was the only language major; all the other students were students from my university in a specialized diplomacy and government service track. Unlike me, they had to pass language proficiency exams, in a language of their choosing, or they did not graduate until they did.
So, I had a great teacher. He did not care about grades at all. He would give bad grades, and when kids would complain give them good ones if they wanted to retest. He would play games with students regarding this. Now the students who wanted good marks to count towards proficiency were really pissed. He put it quite simply: "I'll give you all the A's you want; but when you go back you will still not speak the language. I do not really care and that is not the focus of my classes. I am here to teach and discuss the language and material with you. That is what interests me." I was one of the few that enjoyed that course, and I was admittedly the best student. I continued to be a strong student in all courses, even those I failed (like economics and computer science) because I was far more interested in material than grades.
In the end, I graduated with a high GPA. I see this as a cautionary tale. Now if only I could finally learn C and submit kernel patches like I promised myself in college. But such is life.
Re: To Stop Cheating, Nuclear Officers Ditch the Grades
#24In other words, you get what you measure. This is one of the reasons I discourage measuring team performance in terms of bug counts or lines-of-code. The former lead to what some call "issue-tennis" and informal fixes, and the latter lead to code bloat. After years of trying, I've finally concluded that "key performance indicators" attached to individuals or teams (as opposed to artifacts and periods) usually leads t…
I learned that 30 years ago. It seems to be one of those things that every generation needs to learn the hard way. A reason why an effective team needs a few older guys who've been around the block a few times :-)
I agree. I abhor political talk on HN, but this resonated true with me when hearing the comparison of US troop operations in Iraq to that of Vietnam in those instance. The idea of kill counts, popular in Vietnam, and targeted assasinations by militant leaders in Iraq (made famous by the playing card where people were ranked) and general metrics based on opposition death counts. I reflected on this and wondered how so many officers who survived Vietnam and saw who defining performance metrics the wrong way come back to let the same or even courage it later as officers in Iraq is this recurring horror story in large org management. It scares me that these mistakes are repeated, from war to situations with far less consequence like software development. I am sure there are good psych articles for more details on such issues, and I have always been interested in why such problems recur.
Re: To Stop Cheating, Nuclear Officers Ditch the Grades
#25In other words, you get what you measure. This is one of the reasons I discourage measuring team performance in terms of bug counts or lines-of-code. The former lead to what some call "issue-tennis" and informal fixes, and the latter lead to code bloat. After years of trying, I've finally concluded that "key performance indicators" attached to individuals or teams (as opposed to artifacts and periods) usually leads t…
Here's the obligatory Dilbert -> http://dilbert.com/strips/comic/1995-11-13/
This sounds like a cheating problem, not a measuring problem. The same things happens when people try to measure schools.
My sense is you can't give up on measuring things, because you can't scale an organization without metrics. You just have to balance them with understanding what is really going on. Or rather you need to have the metrics tell us "What's going on with the business" rather than "What's going on with the individual".
Re: To Stop Cheating, Nuclear Officers Ditch the Grades
#26Earlier quoted context omitted.
I agree that KPIs are misused quite a lot but that does change the fact that they can provide valuable information and setup incentives to follow certain goals. I think it is rather simplistic to say that having more than one KPI can be reduced to "a more complicate KPI". Sure, it won't solve all problems with badly setup KPIs but to allow for detailed incentive structures you need detailed KPIs. My major point is st…
> I agree that KPIs are misused quite a lot but that does change the fact that they can provide valuable information and setup incentives to follow certain goals. The problem is that KPIs are misused like, 99.9% of the times, starting from elementary school. It's rare to see someone using such scores as (one of many) evaluation input and not an output. It's good that you put effort into doing real analysis; sadly, no…
Sadly, you are probably right, at least that the majority of cases misuse KPIs. But dont throw the baby out with the bathwater. KPIs reveal patterns, force you attention towards certain issues and create a base for discussion. They can also pidgeon whole you into misunderstanding what is happening though.
In my experience, KPIs work better as a tool for communication and as input for analysis rather than for optimization of goals. In a previous job I worked with aggregating the Balanced Scorecard for a 10,000 employee corporation. On such an aggregate level where the KPIs are composites of many underlying decisions there is less of a risk of perverse incentives. Instead, they are used for comparing trends across time and similar purposes.
KPIs targeted at individuals or smaller teams are very different than that example though.
Re: To Stop Cheating, Nuclear Officers Ditch the Grades
#27So War Games was right then... We really do need to replace the officers with computers.
The Soviets did have an automated system that could launch their missiles automatically: http://en.wikipedia.org/wiki/Dead_Hand_%28nuclear_war%29 It may still be operational, but not normally turned on - for obvious reasons.... At the other end of the spectrum the UK has no automated control preventing unauthorized launches (no Permissive Action Locks) and we check to see whether Radio 4 is still broadcast to detect…
The Air Force was also seriously against permissive action locks. When forced to add them, the generals made the launch codes the same on every warhead, all zeros.
Command and control is very hard: being able to always and immediately launch a nuclear attack but never subject to an unauthorized launch presents a variety of challenges. PALs help with the latter but not the former. Maybe because the Brits have fewer warheads (and therefore the ability to exercise more control) or needed to respond faster (being closer to the USSR and other nuclear powers), they made a strategic choice to skip PALs. Eric Schlosser's book Command and Control is worth a read if you are interested in the history of the US nuclear arsenal.
Re: To Stop Cheating, Nuclear Officers Ditch the Grades
#28We're pretty much guaranteed to have a nuclear accident at this point.
If ditching the grades is the answer to cheating, I say ditch nuclear weapons to solve nuclear problems.
Re: To Stop Cheating, Nuclear Officers Ditch the Grades
#29In other words, you get what you measure. This is one of the reasons I discourage measuring team performance in terms of bug counts or lines-of-code. The former lead to what some call "issue-tennis" and informal fixes, and the latter lead to code bloat. After years of trying, I've finally concluded that "key performance indicators" attached to individuals or teams (as opposed to artifacts and periods) usually leads t…
Here's the obligatory Dilbert -> http://dilbert.com/strips/comic/1995-11-13/ This sounds like a cheating problem, not a measuring problem. The same things happens when people try to measure schools. My sense is you can't give up on measuring things, because you can't scale an organization without metrics. You just have to balance them with understanding what is really going on. Or rather you need to have the metrics…
The problem isn't measurement, it is tying rewards to achieving certain measurements. People will do what you reward them for doing, even if that isn't what you actually wanted them to do.
The real issue is that identifying the correct things to measure and reward is really, really hard.
I read once about a factory that was trying to improve output so they started rewarding shifts of workers for reaching production goals. But to achieve those goals workers started cutting corners and quality went down. So the company installed a device to measure the quality of the widgets, which worked for a while but then quality went down again. The workers had figured out they could just keep measuring the same in-spec widget over and over again to make the machine happy. Next the company altered the quality measurement machine so that it would issue an error if it did two identical measurements in a row. Again quality went up for a while until the workers figured out they could just rotate a couple of good parts through the machine to fool it.
Re: To Stop Cheating, Nuclear Officers Ditch the Grades
#30We're pretty much guaranteed to have a nuclear accident at this point. If ditching the grades is the answer to cheating, I say ditch nuclear weapons to solve nuclear problems.
You go first.