Live data from Hacker News

Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

en.wikipedia.org

11–20 of 137 posts

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#11
post #4

"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.

That is only true if you do that with no regard for the way it was achieved.

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#12
post #11
post #4

"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.

That is only true if you do that with no regard for the way it was achieved.

That's kind of the point the point though. A lot of people will lose focus on the reason for a metric.

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#13
yep, seen several idiots whose work habits are destructive be promoted for gaming stats

seen stats make people discourage/delay income/work to avoid anomalies/spikes in stats because if there's a spike it makes it look like you are slacking the next month, or makes a manager think you should be able to constantly hit the spike

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#14
post #4

"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.

So then you add “size of customer base” to your metrics!

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#16
post #4

"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.

When I interviewed for a data analyst position at a known startup, I was asked to define a target KPI for a controlled pseudo-A/B experiment, and I suggested total revenue. The interviewer replied "that's a dumb answer because revenue can be gamed; you should have said # of sales."

I was too taken aback to follow up asking why # of sales can't be gamed too.

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#17
And as soon as a measure becomes known, it becomes the target.

IMO, the law ought to be "As soon as a measure becomes known it ceases to become a good measure."

"I'm fat because I can't touch my toes? Hang on, I'll be back in three weeks."

"What's that? Your measure for gas consumption and co2 emissions has you go exactly 35 MPH for 10 minutes? Hmm....."

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#19
post #10

I wish more people knew and understood this concept. It's important to know the impact that you have on a system, especially at large scale. Decades after Goodhart made this law, we still have shoddy credit risk measures that put tens of trillions of dollars at risk just in the US alone. Hell, the last recession alone was mostly caused by the government's choice of credit worthiness metrics.

Can you explain how the recession was caused by "government's choice of credit worthiness metrics", instead of rollbacks of regulation?

Re: Goodhart's Law: When a measure becomes a target, it ceases to be a good measure

#20
post #4

"You get what you measure" is the bane of KPIs everywhere. Measure deals closed, and you get deals with no regard for profitability. Measure profitability, and you will burn through your customer base by jacking prices and cutting quality. Etc etc.

So then you add “size of customer base” to your metrics!

Doing so in a large software company would likely result in Sales & Marketing being empowered. Which in turn will likely lead to technical debt (as the company stretches to meet impractical promises), as well as stagnation in Engineering (S&M are delivering on the metrics, so they get promoted).

In general, large organizations are like an AI. In this case, making a goal that won't backfire when applied without thought is really difficult.

Post reply on HN