Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Overfitting and the strong version of Goodhart’s law

sohl-dickstein.github.io

Post reply on HN
Print view View original

31–40 of 111 posts

Re: Overfitting and the strong version of Goodhart’s law

#31

Post by jeffreygoesto »

kqr wrote: post #29
salawat wrote: post #26

Earlier quoted context omitted.

You've now redefined JIT to mean "buffered with the minimum viable buffer" where minimum viable buffer is "whereever I drop the goalposts when defining common-cause variation. It's the definitional version of working out a variable in an equation, making a mistake, and ending up looking at 0=0. Happens all the time.

Yes, but that it literally what JIT means, though. It's a mindset of solving assignable-cause variation at its cause rather than papering over it with more inventory.

Literally, "just" in time would mean "not more (material, goods,...) than needed" in time, wouldn't it? A real world analogy to Kanban.

Re: Overfitting and the strong version of Goodhart’s law

#32

Post by kkoncevicius »

jaschasd wrote: post #17
kqr wrote: post #14

I'm not sure I agree. The problem with overfitting and inability to extrapolate out of sample is variance. The problem with Goodhart's law is bias. I don't think a training sample is to the full population what a proxy metric is to the objective -- not theoretically, and not practically. The training sample faithfully represents the full population (by definition, if it was randomly selected). Any difference in compo…

Author of blog post here! Overfitting can happen in many ways -- your training objective can be different at train and test time, or as you suggest the datapoints you use can be different at train and test time. For overfitting induced by datapoints: If you include the datapoints in your problem specification, then you can say they induce bias at test time. If you treat the choice of training datapoints as a random v…

My first impression is also in agreement with the parent. The blog post appears to use some terms loosely in order to make the connection between overfitting and Goodhart's law stronger. For example - calling training sample "proxy" and stating that it is is a slightly different goal is already leading towards the pre-defined conclusion.

And the reply also leaves me with a similar impression:

> your training objective can be different at train and test time

But this is not overfitting, this is concept drift, a different and well-defined thing in ML.

> the datapoints you use can be different at train and test time

Both train and test data came from the same population. They are just different incomplete random samples.

I guess what I am getting at - overfitting happens because we know we are training a model on an incomplete representation of the whole. But that representation is not a proxy, as suggested in the article - it is not slightly different to the goal. It's an incomplete piece of the goal.

Re: Overfitting and the strong version of Goodhart’s law

#33

Post by lqet »

I recommend reading "The Collapse of Complex Societies" by Joseph Tainter [0].

Complex societies tend to address problems by adding more and more rules and regulations, simply because they have always done so and it has been successful in the past. More importantly, though, it is typically the only tool they have. Essentially these societies are increasingly overfitting their legislature to narrow special cases until it cannot handle anything unexpected anymore. Such a society is highly fragile. I witness this firsthand every day in my own country. Living here feels like lying in a Procrustean bed.

[0] https://www.amazon.com/Collapse-Complex-Societies-Studies-Ar...

Re: Overfitting and the strong version of Goodhart’s law

#34

Post by kqr »

jeffreygoesto wrote: post #31
kqr wrote: post #29

Earlier quoted context omitted.

Yes, but that it literally what JIT means, though. It's a mindset of solving assignable-cause variation at its cause rather than papering over it with more inventory.

Literally, "just" in time would mean "not more (material, goods,...) than needed" in time, wouldn't it? A real world analogy to Kanban.

Kanban is indeed one way to do just in time. It's useful primarily for the more variable flows. For more regular flows, you can often rely on the average flow rate working out and schedule deliveries in advance.

Re: Overfitting and the strong version of Goodhart’s law

#36

Post by c7b »

Not sure I agree. The problem with overfitting is fitting too closely to the data points at hand, but you might still be measuring the right thing, as discussed in other posts here.

The problem with Goodhart's law is, as I've always taken it, closer to the Lucas critique in economics than to the bias-variance trade-off in machine learning. Namely, when it comes to human behavior, structural relations that are very real and present in the training data may break down once you put pressure on them for control purposes.

When you use machine learning to, say, detect skin cancer, you might accidentally learn the markers put into the images to highlight the cancerous region rather than the skin properties - that's overfitting. But the skin cells themselves don't care - they won't alter their behavior whether you detect them correctly (and remove them) or not. If you use a model to find a relation between some input and a human behavior output, humans might very much start to change their behavioral responses once you start to make changes. The entire relation breaks down, even if you've measured it correctly beforehand, because people, unlike particles, have their own interests.

Re: Overfitting and the strong version of Goodhart’s law

#38

Post by legulere »

nine_k wrote: post #5

Everything is a tradeoff. Efficiency is usually a tradeoff that limits flexibility. If you don't happen to need flexibility, you can have amazing efficiency: a wheel of a rail car is so much more efficient than a foot. But only when it stays on a rail. If we look at a somehow more difficult terrain, a foot suddenly happens to be a better deal, because it adapts to a variety of conditions. Of course, at the expense of…

Trade offs are also optimizations: you are optimizing for the sums of weighted advantages and disadvantages of something: https://en.wikipedia.org/wiki/Loss_function

Re: Overfitting and the strong version of Goodhart’s law

#39

Post by TomSwirly »

jaschasd wrote: post #19
whack wrote: post #13

I'm going to go on a rant here. Most times when I hear someone invoke Goodhart's law, it's because they are opposed to transparent and metric-driven decision-making. It's easy to find the flaws with this approach, as the article has effectively done. But what they inevitably ignore is the flaws of not using this approach. Without a transparent metrics-driven decision making process, what you end up with one of the fo…

Blog post author here. Just for the record, I am very much in favor of metric-driven decision making. I suggest some ways we can make it more robust, and also that we should be aware that our metrics may not be measuring what we intend.

This was a great article - I'm sending it to a couple of my friends.

Might I ask what software you used for blogging? I can't seem to find the source repo it came from...

Re: Overfitting and the strong version of Goodhart’s law

#40

Post by kqr »

kkoncevicius wrote: post #32
jaschasd wrote: post #17

Earlier quoted context omitted.

Author of blog post here! Overfitting can happen in many ways -- your training objective can be different at train and test time, or as you suggest the datapoints you use can be different at train and test time. For overfitting induced by datapoints: If you include the datapoints in your problem specification, then you can say they induce bias at test time. If you treat the choice of training datapoints as a random v…

My first impression is also in agreement with the parent. The blog post appears to use some terms loosely in order to make the connection between overfitting and Goodhart's law stronger. For example - calling training sample "proxy" and stating that it is is a slightly different goal is already leading towards the pre-defined conclusion. And the reply also leaves me with a similar impression: > your training objectiv…

I think this is a difficult concept for many without statistical training. The fact that different outcomes can be "the same" from a practical perspective.
Post reply on HN