Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

en.wikipedia.org

Post reply on HN
Print view View original

61–70 of 137 posts

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#61

Post by sonnyblarney »

mikeash wrote: post #52
sonnyblarney wrote: post #42

Earlier quoted context omitted.

When the enemy has limited personnel and equipment - targets such as body count, vehicles destroyed etc. are very good measures of progress. And FYI the US handily and clearly won the Vietnam war; the South Vietnamese insurgency was defeated and the North Korean Army was obliterated. Due to political pressure at home, the US withdrew after the war, and a short while later the North Vietnamese were able to rebuild the…

If “progress” means murdering civilians by the thousand as part of a vain attempt to maintain a vestige of colonialism, count me out.

"If “progress” means murdering civilians by the thousand as part of a vain attempt to maintain a vestige of colonialism, count me out."

Progress was defeating a ruthless and murderous North Korean Communist regime who, upon victory, mass murdered 10's of thousands of their own citizens, put 100's of thousands in concentration camps and ruthless oppressed their population for decades.

My homework partner in high school was a child of Vietnamese immigrants - literally 'boat people'. Also my first boss in the Valley was Vietnamese from that era. They have some stories.

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#62

Post by triviatise »

We hold people accountable for actions instead of outcomes, since outcomes (even profit) can be gamed at the overall detriment of the organization.

We talk about the outcome we want, look at the levers that will get the outcome, and then agree to the actions that we are going to take.

If the actions dont result in the forecasted outcome then the underlying assumptions are wrong. The assumptions are wrong if 1) people arent doing the tasks properly 2) the relationship between the task and the outcome are different than what we thought. If the issue is 1, we provide training. If the issue is 2, we use the revised assumption for future forecasts.

We dont use the word target anymore, we use the word forecast.

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#64

Post by njarboe »

nkoren wrote: post #55
njarboe wrote: post #34

The breaking of this law when one uses profit as the measure of a successful company or business is why I think that a well set up market system actually works. People and companies can set up making a profit as their target and it is still a good measure of success, because it is actually what you want. The key for a good society is regulating the places where making max profits can induce bad behavior and leaving a…

Even disregarding the potential for negative externalities etc., profit alone can still be a terrible KPI. I worked for a large engineering consultancy that was very profit motivated. They figured out a brilliant way to maximise profit: get rid of business development! Only pay for work when clients are paying you to do it. This was GREAT for quarterly profit margins, at the expense of the project pipeline. As the pi…

I would say the complexity is in balancing profit over time. Here is your contradictory force. In the end that large engineering consultancy was making zero profit and was worth nothing. Whether running a business into the ground will give you more overall profit than just selling it and putting the money into government bonds is something one can consider, but in your case was the consultancy owned by the people making these short-term profit maximizing decisions? The agent/principal problem for companies has not been solved.

It is probably not a coincidence that the top 6 US companies by market cap (Apple, Amazon, Alphabet, Microsoft, Facebook, Berkshire Hathaway) are still run by their founders or were run by their founders for most of the companies existence before a recent step-down (Microsoft) or death (Apple). Most public companies aren't and they struggle to grow or even survive over time.

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#65

Post by triviatise »

logicallee wrote: post #26

Aren't there are exceptions? For example, a measure of health might be life expectancy. If this becomes known, is it no longer a good measure? I think it's still a good measure. I mean, how can you get to a life expectancy of 100 without a healthy population?

Life expectancy is actually a really great example of the opposite. People routinely claim our health system is bad compared to europe because our average life expectancy is lower.

It turns out african americans skew our metrics because they have

1) high infant mortality 2) high murder rates of young men

The high infant mortality is irrespective of wealth and has a disproportionate impact on average life expectancy because the death of a 0 age person has a huge impact on the average. When you look at the rest of the population, our whites are on par with europe and our asians are on par with asian countries.

One way this is gamed is that some countries have different standards for reporting infant mortality.

>

In the US it is gamed in reverse in the sense that the average hides the true problem is with subpopulations and not the entire US medical system as a whole. Our (gamed) poor performance in this statistic is used as a measure to support moving to a substantially different system.

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#66

Post by mlthoughts2018 »

maxxxxx wrote: post #9
smallnamespace wrote: post #6

Earlier quoted context omitted.

You shouldn't optimize at all but instead pick multiple, 'distantly related' measures and satisfice over them. That's sort of what humans do to have a robust objective function. Sufficient food, water, physical safety, love, social interaction, meaningful work, spiritual fulfillment—fulfill each of them and humans generally move on to the next 'unfulfilled' goal on the list rather than endlessly maximizing something…

Exactly. Don't reduce everything down to one factor. In business you need profit AND customer retention AND ongoing development AND some level of luck AND a lot of other factors. Same for health. You need enough sleep AND good food AND a good mental state AND good genes AND a lot of other things. Focusing on only one thing while neglecting others doesn't work.

But then your "one factor" just becomes a combination of various other factors. I don't think the point of Goodhart's Law is related to "over-optimizing for just one thing," rather it's a statement about whether any metric (even a broad, diverse metric that takes into account multiple different things) can retain informativeness once it is set up as an optimization target.

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#67

Post by Semirhage »

cwmma wrote: post #56
Semirhage wrote: post #43

Earlier quoted context omitted.

And FYI the US handily and clearly won the Vietnam war; the South Vietnamese insurgency was defeated and the North Korean Army was obliterated. That’s a serious, if delusional claim that requires a loooooot of support. “Handily” in particular seems like a ridiculous way to state it, even if you only look at US casualties.

There's a quote to the extent of "the US never lost a battle in the Vietnam, but still lost the war" which is probably what they were getting at, which just honestly proves the point that America was really optimizing for the wrong measure in that war.

I respect your charitable interpretation, but it turns out to be overly charitable based on subsequent posts. I’m going to take a page out of the “know when to back away slowly” handbook, and back away slowly from this train wreck.

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#68

Post by Semirhage »

sonnyblarney wrote: post #61
mikeash wrote: post #52

Earlier quoted context omitted.

If “progress” means murdering civilians by the thousand as part of a vain attempt to maintain a vestige of colonialism, count me out.

"If “progress” means murdering civilians by the thousand as part of a vain attempt to maintain a vestige of colonialism, count me out." Progress was defeating a ruthless and murderous North Korean Communist regime who, upon victory, mass murdered 10's of thousands of their own citizens, put 100's of thousands in concentration camps and ruthless oppressed their population for decades. My homework partner in high schoo…

“Progress was defeating a ruthless and murderous North Korean Communist regime...”

“Who, upon victory, mass murdered 10's of thousands of their own citizens...”

Proofread! Or at the very least, try not to include outright contradictions in your outlandish claims. Claiming that the US won in one sentence, and in the very next declaring the opposite is baffling.

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#69

Post by yongjik »

sonnyblarney wrote: post #61
mikeash wrote: post #52

Earlier quoted context omitted.

If “progress” means murdering civilians by the thousand as part of a vain attempt to maintain a vestige of colonialism, count me out.

"If “progress” means murdering civilians by the thousand as part of a vain attempt to maintain a vestige of colonialism, count me out." Progress was defeating a ruthless and murderous North Korean Communist regime who, upon victory, mass murdered 10's of thousands of their own citizens, put 100's of thousands in concentration camps and ruthless oppressed their population for decades. My homework partner in high schoo…

You might want to know that the South Korean regime, under the blessing (or indifference) of USA, had no problem mass murdering 100k~200k of its own people for being communists.[1] Another ~60k people were murdered in the island of Jeju to "put down" communist insurgents, where entire towns were wiped out.[2] And during the Korean war they managed to kill another +100k by letting conscripts starve or freeze through corruption.[3]

History is never about "good guys killing bad guys".

[1] https://en.wikipedia.org/wiki/Bodo_League_massacre

[2] https://en.wikipedia.org/wiki/Jeju_uprising

[3] https://en.wikipedia.org/wiki/National_Defense_Corps_inciden...

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#70

Post by maxxxxx »

mlthoughts2018 wrote: post #66
maxxxxx wrote: post #9

Earlier quoted context omitted.

Exactly. Don't reduce everything down to one factor. In business you need profit AND customer retention AND ongoing development AND some level of luck AND a lot of other factors. Same for health. You need enough sleep AND good food AND a good mental state AND good genes AND a lot of other things. Focusing on only one thing while neglecting others doesn't work.

But then your "one factor" just becomes a combination of various other factors. I don't think the point of Goodhart's Law is related to "over-optimizing for just one thing," rather it's a statement about whether any metric (even a broad, diverse metric that takes into account multiple different things) can retain informativeness once it is set up as an optimization target.

I think you could set up an optimization target for a company that says "be profitable and grow for the next 30 years". You would see quickly that this target is a complex beast with a lot of different factors at play that need to be balanced against each other. I don't think Goodhart's law would apply here.
Post reply on HN