People respond more or less rationally to their incentives. Who knew?
Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
91–100 of 137 posts
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#92Earlier quoted context omitted.
When I interviewed for a data analyst position at a known startup, I was asked to define a target KPI for a controlled pseudo-A/B experiment, and I suggested total revenue. The interviewer replied "that's a dumb answer because revenue can be gamed; you should have said # of sales." I was too taken aback to follow up asking why # of sales can't be gamed too.
Credit where due, I think this is the first time I've read of anybody being told how they got something wrong in an interview.
Why is that credit? Good interview questions don't have a right or wrong answer, they have a right or wrong process of coming up with an answer. Especially for a question like "Pick a metric," almost any relevant metric is defensible and has tradeoffs, and the question should be about how the interviewee analyzes engineering tradeoffs, gathers requirements, defends their initial proposal, and is open to counterarguments.
(If there's a correct answer, you can just give them a quiz. If there's a Googleable correct answer, you can just employ them and give them Google.)
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#93Earlier quoted context omitted.
That seems to happen in all wars. Aircraft downed and tanks destroyed in WWII, for example. There were occasions in the Vietnam War where regions were declared to be free fire zones, and everyone within was to be killed, down to the last child and granny.
in principle, you can verify every downed aircraft and destroyed tank with a photograph or a video. it's hard to move the wrecks of each. there are worse metrics.
But still, such figures as reported were inflated by large margins (something on the order of 25 to 100 percent) on both the Allied and Axis sides.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#94So we should minimize the fraction of measures which have become targets then?
I love the irony in this.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#95The same principle applies to attributes and qualities which mark successful entrepreneurs. The same principle applies to attributes and qualities which mark good developers. The same principle applies to attributes and qualities which mark good employees. Good people. Good politicians. Activists. Good pundits. Intellectuals. Top tier school degrees.
Hacker News karma.
Shouldn't we all be looking out for the "targeting" and be trying to find the next measure?
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#96Earlier quoted context omitted.
National Security Action Memoranda number 52, May 1961: https://www.jfklibrary.org/Asset-Viewer/Archives/JFKNSF-330-... Subject: Program of Action to Prevent Communist Domination of South Viet Nam The US Objective: to prevent Communist domination of South Vietnam and to create in that country a viable and increasingly democratic society. The US had not achieved those objectives 12 years later. The communists won cont…
"The US had not achieved those objectives 12 years later." Yes. They did. The US did achieve those objectives in the early 1970s'. The US vanquished the South Vietnamese forces, and did considerably damage to North Vietnamese forces using air power - to the point where North Vietnamese did not have a capable fighting force. Communism was basically gone in the South, in the North, inoperable, at least militarily. Then…
If the North didn't have a capable fighting force any more, and still overwhelmed the South shortly after the US withdrew, that means the US did not actually establish a "a viable and increasingly democratic society" during the time it had a military presence there. Societal viability includes, at a minimum, being able to survive one's enemies. Enemies that were already drastically weakened during the period of US involvement, by your own account.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#97"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.
That’s why organisations need leaders who can change the direction having realised that the current state had degraded to being gamed. Seek revenue, then seek new clients but not revenue, then go on R&D spree. People are not going to be happy, but the organisation is going to be healthy. IMO there must be high-level metrics which define which low-level metrics are valuable today.
It's getting weirder when you factor in shareholder value: sacrificing the future of the company for looking good in all the fashionable metrics might be exactly the leaders job, depending on how the board is made up.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#98Earlier quoted context omitted.
"The US had not achieved those objectives 12 years later." Yes. They did. The US did achieve those objectives in the early 1970s'. The US vanquished the South Vietnamese forces, and did considerably damage to North Vietnamese forces using air power - to the point where North Vietnamese did not have a capable fighting force. Communism was basically gone in the South, in the North, inoperable, at least militarily. Then…
If the North didn't have a capable fighting force any more, and still overwhelmed the South shortly after the US withdrew, that means the US did not actually establish a "a viable and increasingly democratic society" during the time it had a military presence there. Societal viability includes, at a minimum, being able to survive one's enemies. Enemies that were already drastically weakened during the period of US in…
Societal viability includes, at a minimum, being able to survive one's enemies.
This needs to be said a lot more in our civilization, in 2018.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#99Earlier quoted context omitted.
"The US had not achieved those objectives 12 years later." Yes. They did. The US did achieve those objectives in the early 1970s'. The US vanquished the South Vietnamese forces, and did considerably damage to North Vietnamese forces using air power - to the point where North Vietnamese did not have a capable fighting force. Communism was basically gone in the South, in the North, inoperable, at least militarily. Then…
If the North didn't have a capable fighting force any more, and still overwhelmed the South shortly after the US withdrew, that means the US did not actually establish a "a viable and increasingly democratic society" during the time it had a military presence there. Societal viability includes, at a minimum, being able to survive one's enemies. Enemies that were already drastically weakened during the period of US in…
Societal viability includes, at a minimum, being able to survive one's enemies.
By that argument, all of western Europe didn't save a viable society since Germany was able to defeat them.
I don't agree with that.
Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure
#100And as soon as a measure becomes known, it becomes the target. IMO, the law ought to be "As soon as a measure becomes known it ceases to become a good measure." "I'm fat because I can't touch my toes? Hang on, I'll be back in three weeks." "What's that? Your measure for gas consumption and co2 emissions has you go exactly 35 MPH for 10 minutes? Hmm....."
Gamification increases as understanding improves. Information is exploited.
[Targeting Intensifies]