Without dates and metrics you can’t measure progress. Like mile markers along the road (back before GPS / Smartphones). But without continuous, high trust, collaborative conversation you can’t actually make progress. That’s the gasoline powering you down the road. The goal should be Disneyland (shouldn’t it always be Disneyland?), not the next mile marker.
Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
21–30 of 137 posts
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#22And as soon as a measure becomes known, it becomes the target. IMO, the law ought to be "As soon as a measure becomes known it ceases to become a good measure." "I'm fat because I can't touch my toes? Hang on, I'll be back in three weeks." "What's that? Your measure for gas consumption and co2 emissions has you go exactly 35 MPH for 10 minutes? Hmm....."
> And as soon as a measure becomes known, it becomes the target.
Soon after, 90% of the target is still an 'A', which becomes the new target.
On a related note of unintended consequences, the Cobra Effect: https://en.wikipedia.org/wiki/Cobra_effect
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#23(cough! cough! "scrum")
(cough! cough! "agile")
(cough! cough! "daily stand-ups")
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#24"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.
That is only true if you do that with no regard for the way it was achieved.
That’s technically correct but it’s like saying we shouldn’t have an obesity crisis because anyone who eats less than they burn will reliably lose weight. We have too much evidence about human nature to treat something which happens so commonly as an edge case.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#25During the Vietnam War, it was body count. That turned out to be a great way of incentivizing the most murderous troops.
Probably more of an incentive for NCO's and Officers to overclaim the results, so they look good to their superiors.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#26Aren't there are exceptions? For example, a measure of health might be life expectancy. If this becomes known, is it no longer a good measure? I think it's still a good measure. I mean, how can you get to a life expectancy of 100 without a healthy population?
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#27This is probably where I should mention my two longer articles that discuss this in depth;
https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why-...
https://www.ribbonfarm.com/2016/09/29/soft-bias-of-underspec...
As well as my more recent paper categorizing the different ways it occurs; https://arxiv.org/abs/1803.04585
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#28During the Vietnam War, it was body count. That turned out to be a great way of incentivizing the most murderous troops.
Probably more of an incentive for NCO's and Officers to overclaim the results, so they look good to their superiors.
That seems to happen in all wars. Aircraft downed and tanks destroyed in WWII, for example.
There were occasions in the Vietnam War where regions were declared to be free fire zones, and everyone within was to be killed, down to the last child and granny.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#29So we should minimize the fraction of measures which have become targets then?
You shouldn't optimize at all but instead pick multiple, 'distantly related' measures and satisfice over them. That's sort of what humans do to have a robust objective function. Sufficient food, water, physical safety, love, social interaction, meaningful work, spiritual fulfillment—fulfill each of them and humans generally move on to the next 'unfulfilled' goal on the list rather than endlessly maximizing something…
That's often helpful, but not sufficient in general. See the dynamics discussed in my article here; https://arxiv.org/abs/1803.04585
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#30"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.
That’s why organisations need leaders who can change the direction having realised that the current state had degraded to being gamed.
Seek revenue, then seek new clients but not revenue, then go on R&D spree. People are not going to be happy, but the organisation is going to be healthy.
IMO there must be high-level metrics which define which low-level metrics are valuable today.