Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
1–10 of 137 posts
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#2I’ve seen this a lot. Even people who intellectually know that correlation doesn’t equal causality make this mistake all the time.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#3So we should minimize the fraction of measures which have become targets then?
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#4"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#5I’ve seen this a lot. Even people who intellectually know that correlation doesn’t equal causality make this mistake all the time.
It's not even that correlation doesn't equal causality, it's that the correlation may also completely reverse.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#6So we should minimize the fraction of measures which have become targets then?
You shouldn't optimize at all but instead pick multiple, 'distantly related' measures and satisfice over them.
That's sort of what humans do to have a robust objective function. Sufficient food, water, physical safety, love, social interaction, meaningful work, spiritual fulfillment—fulfill each of them and humans generally move on to the next 'unfulfilled' goal on the list rather than endlessly maximizing something that ceases to improve their life quality.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#7Indeed this is the whole idea behind OKR. The Objectives are _supposed_ to be grand, audacious, and only achievable under extreme stretching. Furthermore, they are _supposed_ to not be used as a KPI for performance reviews.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#8I've seen similar topics come up on HN front page every once in a while.
Here are some of my thoughts on the topic, through the prism of metrics for a company.
http://dimitarsimeonov.com/2018/03/23/the-vanity-metric-para...
TLDR of my post: The vanity metric paradox - every metric, sufficiently optimized becomes a vanity metric, as it stops being the make-or-break qualifier for a company.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#9So we should minimize the fraction of measures which have become targets then?
You shouldn't optimize at all but instead pick multiple, 'distantly related' measures and satisfice over them. That's sort of what humans do to have a robust objective function. Sufficient food, water, physical safety, love, social interaction, meaningful work, spiritual fulfillment—fulfill each of them and humans generally move on to the next 'unfulfilled' goal on the list rather than endlessly maximizing something…
Exactly. Don't reduce everything down to one factor.
In business you need profit AND customer retention AND ongoing development AND some level of luck AND a lot of other factors.
Same for health. You need enough sleep AND good food AND a good mental state AND good genes AND a lot of other things.
Focusing on only one thing while neglecting others doesn't work.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#10I wish more people knew and understood this concept.
It's important to know the impact that you have on a system, especially at large scale. Decades after Goodhart made this law, we still have shoddy credit risk measures that put tens of trillions of dollars at risk just in the US alone. Hell, the last recession alone was mostly caused by the government's choice of credit worthiness metrics.