Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Overfitting and the strong version of Goodhart’s law

sohl-dickstein.github.io

Post reply on HN
Print view View original

81–90 of 111 posts

Re: Overfitting and the strong version of Goodhart’s law

#81

Post by kqr »

marcosdumay wrote: post #67
jeffreygoesto wrote: post #31

Earlier quoted context omitted.

Literally, "just" in time would mean "not more (material, goods,...) than needed" in time, wouldn't it? A real world analogy to Kanban.

JIT is mostly a name for inter-companies Kanban (the factory version of Kanban, not the software one). And because of that, yes, the amount of inventory is pretty much arbitrary.

This is a common misconception. People often talk about the kanban style JIT because it looks impressive, in some sense. There's all these things going back and forth.

But as Taiichi Ohno himself would tell you: if it looks impressive, it's wasteful. It's a lot of motion with little movement. Efficiency looks disappointingly obvious and simple.

Kanban is a necessary evil when you have been unsuccessful in driving out variation, it's not a desirable state of things.

Re: Overfitting and the strong version of Goodhart’s law

#82

Post by AnthonyMouse »

club_tropical wrote: post #45

I have mentioned before - I hate Goodhart's law. It makes no sense. There is no example that is a good measure but a bad target. The canonical example I have heard is hospital emergency rooms that started to be measured by wait times, so they refused to admit patients until staff was ready to receive them, literally having ambulances circling around the block. This was supposed to be a "good measure turned into bad t…

Do tell what is a specific example of a good measure which is not a bad target.

Re: Overfitting and the strong version of Goodhart’s law

#83

Post by AnthonyMouse »

You have to be careful here though, because the "solutions" can be susceptible to the same process, e.g.:

> Use a progressive tax code, so that unusual success is linked to disproportionately greater cost

A progressive tax code is an increase in complexity and is what leads to arbitrage opportunities and multinational corporations in practice paying lower rates than small and medium domestic businesses. Notice that we already have a "progressive tax code" and it hasn't worked.

A better (simpler) solution is to combine a flat tax rate (e.g. VAT) with a UBI, which produces the effective rate curve you want while being harder for megacorps to avoid because they can't change the location of their customers.

On the other hand, this one is likely to actually work:

> Develop as many complex, inscrutable, and diverse market trading instruments as possible, vesting on as many timescales as possible. (In nature, more complex ecosystems are more stable. Maybe there is a parallel for markets?)

Because, somewhat counterintuitively, what you want is the combination of regulatory simplicity and regulatory diversity. In other words, every place has simple rules, but every place has different rules, which prevents monoculture.

The last thing you want is complex regulation imposed centrally, as it prevents anything from out-competing it when it goes wrong until it goes very wrong.

Re: Overfitting and the strong version of Goodhart’s law

#84

Post by cs702 »

jaschasd wrote: post #78
cs702 wrote: post #73

Except that AI models, especially large deep ones, do NOT overfit like the author thinks. They exhibit what is now called "deep double descent" -- the validation error declines, then increases, and then declines again : https://openai.com/blog/deep-double-descent/ A question I've pondered for a while is whether complex systems in the real world also exhibit double descent. For example, transitioning an online applica…

Blog post author here. A brief note that I do discuss the deep double descent phenomenon in the blog. See the section starting with "One of the best understood causes of extreme overfitting is that the expressivity of the model being trained too closely matches the complexity of the proxy task." I avoided using the actual term double descent, since I thought it would add unnecessary complexity. Lesson learned for nex…

Thank you.

As you probably know, the big deal about double descent is that once sufficiently large AI models cross the so-called "interpolation threshold" in training, and get over the hump, they start generalizing better -- the opposite of overfitting. State-of-the-art performance in fact requires getting over the hump. As far as I can tell, you did not mention any of that explicitly anywhere in your post.

Also, all your plots show only the classical overfitting curve, not the actual curve we now see all the time with larger AI models like Transformers.

Re: Overfitting and the strong version of Goodhart’s law

#85

Post by jaschasd »

cs702 wrote: post #84
jaschasd wrote: post #78

Earlier quoted context omitted.

Blog post author here. A brief note that I do discuss the deep double descent phenomenon in the blog. See the section starting with "One of the best understood causes of extreme overfitting is that the expressivity of the model being trained too closely matches the complexity of the proxy task." I avoided using the actual term double descent, since I thought it would add unnecessary complexity. Lesson learned for nex…

Thank you. As you probably know, the big deal about double descent is that once sufficiently large AI models cross the so-called "interpolation threshold" in training, and get over the hump, they start generalizing better -- the opposite of overfitting. State-of-the-art performance in fact requires getting over the hump. As far as I can tell, you did not mention any of that explicitly anywhere in your post. Also, all…

It's true that I don't go into detail about double descent, though I do describe how increasing capacity often reduces overfitting.

I believe the figure labeled "Figure 1" illustrates what your are suggesting (despite being labeled Figure 1, it is actually at the bottom of the blog post, so maybe easy to miss).

Re: Overfitting and the strong version of Goodhart’s law

#86

Post by epgui »

visarga wrote: post #23
epgui wrote: post #11

I think there are some similarities between Goodhart’s law and overfitting, but I don’t think the lesson or underlying mechanism or model is the same. Crudely speaking, Goodhart’s law is a reflection of the system reacting to an intervention (ie.: dynamic feedback loops) and that just has nothing to do with overfitting.

Training a neural net is a dynamic feedback loop too. Back-propagation is the feedback phase.

Not in the same sense whatsoever. Training a neural net, backpropagation or not, doesn't affect the data. It's basically just some variation of / remix of a linear regression.

Re: Overfitting and the strong version of Goodhart’s law

#87

Post by cs702 »

jaschasd wrote: post #85
cs702 wrote: post #84

Earlier quoted context omitted.

Thank you. As you probably know, the big deal about double descent is that once sufficiently large AI models cross the so-called "interpolation threshold" in training, and get over the hump, they start generalizing better -- the opposite of overfitting. State-of-the-art performance in fact requires getting over the hump. As far as I can tell, you did not mention any of that explicitly anywhere in your post. Also, all…

It's true that I don't go into detail about double descent, though I do describe how increasing capacity often reduces overfitting. I believe the figure labeled "Figure 1" illustrates what your are suggesting (despite being labeled Figure 1, it is actually at the bottom of the blog post, so maybe easy to miss).

> It's true that I don't go into detail about double descent, though I do describe how increasing capacity often reduces overfitting.

I agree.

> I believe the figure labeled "Figure 1" illustrates what your are suggesting (despite being labeled Figure 1, it is actually at the bottom of the blog post, so maybe easy to miss).

Easy to miss, yes. I'm not sure it illustrates the phenomenon, though. That plot shows extreme overfitting (i.e., interpolation) by the 10,000 parameter model. No one really understands what actually happens after interpolation. There's in fact some anecdotal evidence that after crossing the interpolation threshold, large AI models trained with SGD gradually begin to ignore outliers and find simpler models (!) that generalize better (!). Counterintuitive, I know. This is an active area of research, with no good explanations yet, AFAIK.

Re: Overfitting and the strong version of Goodhart’s law

#88

Post by c7b »

jaschasd wrote: post #68
c7b wrote: post #36

Not sure I agree. The problem with overfitting is fitting too closely to the data points at hand, but you might still be measuring the right thing, as discussed in other posts here. The problem with Goodhart's law is, as I've always taken it, closer to the Lucas critique in economics than to the bias-variance trade-off in machine learning. Namely, when it comes to human behavior, structural relations that are very re…

A note that the datapoints you train on are part of the training objective. If you are using different data at test time than you use at training time, then you are measuring the wrong thing during training, the same as if you used a different loss function at training time. Also -- as you say, feedback loops and non-stationarity make everything more complex, and are ubiquitous in the real world! But in machine learn…

Cool that you're responding here. Well, regarding robotics, I'm sure there's all sorts of problems when it comes to training models, but I'm not sure that Goodhart's law is one of them, unless you can give a concrete example. It's really geared towards social problems. Sure, some natural systems may also exhibit the kind of adaptive response that leads to the breakdown of structural relattions (eg the cancer cells mentioned before may evolve to avoid detection by the AI), but that happens on completely different timescales.

Re: Overfitting and the strong version of Goodhart’s law

#89

Post by [deleted] »

AnthonyMouse wrote: post #80
mannykannot wrote: post #64

Earlier quoted context omitted.

We have already moved on, and a case like this would no longer be regarded as unusual, let alone difficult. In the process, something like what you propose arose, in the form of liability insurance. The system works reasonably well, in that there are a lot fewer hard cases without creating a crippling burden in the average case. If anything, this case, together with the way things have changed since then, demonstrate…

But that only moves the problem somewhere else, because insurance is a moral hazard. Someone who would otherwise be cautious to prevent harm has less incentive to do it because when the harm comes the insurance pays.

[deleted]

Re: Overfitting and the strong version of Goodhart’s law

#90

Post by jaschasd »

cs702 wrote: post #87
jaschasd wrote: post #85

Earlier quoted context omitted.

It's true that I don't go into detail about double descent, though I do describe how increasing capacity often reduces overfitting. I believe the figure labeled "Figure 1" illustrates what your are suggesting (despite being labeled Figure 1, it is actually at the bottom of the blog post, so maybe easy to miss).

> It's true that I don't go into detail about double descent, though I do describe how increasing capacity often reduces overfitting. I agree. > I believe the figure labeled "Figure 1" illustrates what your are suggesting (despite being labeled Figure 1, it is actually at the bottom of the blog post, so maybe easy to miss). Easy to miss, yes. I'm not sure it illustrates the phenomenon, though. That plot shows extreme…

(the plot shows extreme overfitting with a 10 parameter model, and interpolation with a 10,000 parameter model)
Post reply on HN