Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Overfitting and the strong version of Goodhart’s law

sohl-dickstein.github.io

Post reply on HN
Print view View original

91–100 of 111 posts

Re: Overfitting and the strong version of Goodhart’s law

#91

Post by brookst »

marcosdumay wrote: post #66
brookst wrote: post #4

This keeps getting rediscovered in new domains. JIT was the savior of manufacturing, until people learned that a single traffic jam that delayed a single delivery could create costs far in excess of the inventory savings. Optimizations are critical, everywhere. And measuring optimizations is important because it is is easier and cheaper and earlier than measuring end results. Measuring days of inventory, or dollars i…

I highly recommend playing the beer game with different inventory sizes, and looking at the results. Inventory management is not a simple task, and can not be generalized like this. JIT was adopted because it reduced the number of supply chain disasters, not despite increasing it like you claim. But, of course, that reduction wasn't homogeneous and not every single place saw a gain.

I apologize for the unclarity. I was not trying to assert that JIT is fundamentally wrong and always leads to disaster.

My point was that any optimization process can go wrong when people start focusing on maximizing the optimization rather than the ultimate goal. That is how JIT goes wrong. It is also how overfitting appears in neural net training, how security flaws appear in branch prediction, etc.

I'm an MBA. I love JIT. But it's undeniable that JIT has led to disasters. I'm not blaming the approach, I'm blaming specific implementations.

Re: Overfitting and the strong version of Goodhart’s law

#92

Post by cs702 »

jaschasd wrote: post #90
cs702 wrote: post #87

Earlier quoted context omitted.

> It's true that I don't go into detail about double descent, though I do describe how increasing capacity often reduces overfitting. I agree. > I believe the figure labeled "Figure 1" illustrates what your are suggesting (despite being labeled Figure 1, it is actually at the bottom of the blog post, so maybe easy to miss). Easy to miss, yes. I'm not sure it illustrates the phenomenon, though. That plot shows extreme…

(the plot shows extreme overfitting with a 10 parameter model, and interpolation with a 10,000 parameter model)

Interpolation == extreme overfitting.

Double descent phenomenon is what happens after interpolation.

--

RESPONDING TO YOUR LAST COMMENT (after reaching thread depth limit):

Think of it this way: Why and how does the model's performance continue to improve on previously unseen samples after the model has fully overfit (interpolated between) all training samples? Interpolation is not the end-point in training, but a temporary threshold after which models learn to generalize better, improving on interpolation. How is it that these models improve on interpolation?

Re: Overfitting and the strong version of Goodhart’s law

#93

Post by mannykannot »

AnthonyMouse wrote: post #80
mannykannot wrote: post #64

Earlier quoted context omitted.

We have already moved on, and a case like this would no longer be regarded as unusual, let alone difficult. In the process, something like what you propose arose, in the form of liability insurance. The system works reasonably well, in that there are a lot fewer hard cases without creating a crippling burden in the average case. If anything, this case, together with the way things have changed since then, demonstrate…

But that only moves the problem somewhere else, because insurance is a moral hazard. Someone who would otherwise be cautious to prevent harm has less incentive to do it because when the harm comes the insurance pays.

Insurance is not free, and it generally gets more expensive the more reckless you are (or seem) to be (which is not to say this is the only factor in whether this is a zero-sum game, but it is arguably the most objectively quantifiable.)

Re: Overfitting and the strong version of Goodhart’s law

#94

Post by visarga »

epgui wrote: post #86
visarga wrote: post #23

Earlier quoted context omitted.

Training a neural net is a dynamic feedback loop too. Back-propagation is the feedback phase.

Not in the same sense whatsoever. Training a neural net, backpropagation or not, doesn't affect the data. It's basically just some variation of / remix of a linear regression.

Yes, for that you need RL. An environment beats a fixed, even large, training set.

Re: Overfitting and the strong version of Goodhart’s law

#95

Post by jaschasd »

jaschasd wrote: post #90
cs702 wrote: post #87

Earlier quoted context omitted.

> It's true that I don't go into detail about double descent, though I do describe how increasing capacity often reduces overfitting. I agree. > I believe the figure labeled "Figure 1" illustrates what your are suggesting (despite being labeled Figure 1, it is actually at the bottom of the blog post, so maybe easy to miss). Easy to miss, yes. I'm not sure it illustrates the phenomenon, though. That plot shows extreme…

(the plot shows extreme overfitting with a 10 parameter model, and interpolation with a 10,000 parameter model)

I can't reply directly -- is there a maximum thread depth, or a maximum conversation depth?

Anyway -- I wanted to apologize for misreading -- I missed the parenthetical "interpolation" in your comment. I think we are both interpreting the plot the same way.

In terms of your comment about anecdotal evidence -- are you talking about the case where data and model size are increased jointly? If so, I agree, though I don't think that is any longer cleanly to do with double descent/overparameterization.

Re: Overfitting and the strong version of Goodhart’s law

#96

Post by AnthonyMouse »

mannykannot wrote: post #93
AnthonyMouse wrote: post #80

Earlier quoted context omitted.

But that only moves the problem somewhere else, because insurance is a moral hazard. Someone who would otherwise be cautious to prevent harm has less incentive to do it because when the harm comes the insurance pays.

Insurance is not free, and it generally gets more expensive the more reckless you are (or seem) to be (which is not to say this is the only factor in whether this is a zero-sum game, but it is arguably the most objectively quantifiable.)

The moral hazard is the difference between the insurance payout and the amount your future rates change as a result of the claim. If this amount is zero, there is no insurance. If it isn't, there is that much less incentive to avoid the harm.

Re: Overfitting and the strong version of Goodhart’s law

#97

Post by Aicy »

Filligree wrote: post #58
agalunar wrote: post #54

Earlier quoted context omitted.

Ironically, we're not very efficient at all with our land usage today. Nearly 60% of global agricultural land is used for beef (either for pasture or for growing feed) and yet it accounts for only 2% of our calories. And about 50% of all food is wasted – simply thrown away.

I doubt it affects your central claim, but do note that a lot of pasture land is unusable for anything else.

Unfortunately, in practice, the opposite is true.

We grow food, such as grains, just to feed it to cattle which has 1% of the calorie efficiency. For example in the EU 63% of arable land is used to grow animal feed rather than food directly for humans.

Re: Overfitting and the strong version of Goodhart’s law

#98

Post by User23 »

AnthonyMouse wrote: post #80
mannykannot wrote: post #64

Earlier quoted context omitted.

We have already moved on, and a case like this would no longer be regarded as unusual, let alone difficult. In the process, something like what you propose arose, in the form of liability insurance. The system works reasonably well, in that there are a lot fewer hard cases without creating a crippling burden in the average case. If anything, this case, together with the way things have changed since then, demonstrate…

But that only moves the problem somewhere else, because insurance is a moral hazard. Someone who would otherwise be cautious to prevent harm has less incentive to do it because when the harm comes the insurance pays.

Not necessarily. In the case of accidents, liability insurance presents no moral hazard. The ordinary person is not going to leave a broken railing in his house that could cause a guest to fall and injure himself just because he has homeowner's insurance.

Similarly, the ordinary person doesn't seek out automobile accidents on account of having mandatory automobile insurance.

Perhaps most strikingly, the moral hazard theory would suggest that life insurance policy holders are more likely to commit suicide, but in reality they are less likely to![1]

Sure, there are probably vanishingly rare exceptions to the above, but the moral hazard concern is wildly overblown for consumer insurance products purchased by ordinary people.

[1] https://www.munichre.com/us-life/en/perspectives/suicide-mor...

Re: Overfitting and the strong version of Goodhart’s law

#99

Post by Aicy »

bigfudge wrote: post #44

A lot of the suggested mitigations made a lot of sense to me. For example reducing time to prepare grant applications, or to add 1s jitter to stock trades. Perhaps there are flaws to these specific ideas, but the thing that struck me most was that we have almost no way of implementing ideas like this in Anglo political countries right now. Entrenched interests, and the technical detail involved makes it hard to imagi…

> For example reducing time to prepare grant applications, or to add 1s jitter to stock trades.

The 1s jitter idea was actually moderately successful in the game Grepolis, the successor to Tribalwars, but not perfect. Tribalwars is a browser based MMO RTS game, where a typical game takes multiple months, and a typical attack may take anywhere because 20 minutes and 12 hours (in real time) to reach it's target.

When you sent an attack in Tribalwars it'd have a timer to it, but the battle itself would happen instantaneously. A normal attack just steals whatever resources are available in the village being attacked. In order to conquer someone else's village you must have a successful attack with a "noble" in it, which is a very expensive and slow unit. Attacks can only be a certain size (limited to the population of each village), so in order to take a city you would launch many "clearing" attacks at once at the village you want to conquer, aiming for them to battle slightly earlier than your noble attack, to clear any defences beforehand.

This led to the phenomenon that the best way to defend your village was to leave your village completely defenceless for the initial n-1 attacks clearing, and then slip all your defences back home before the nth (noble) attack arrived.

This started with players optimising their defence to arrive 1 second before the noble attack arrived. However the attacking players adapted, but would perfectly launch attacks based on travel times such that they would all launch in the same second, but maybe 100ms seconds off. This resulted in players using third party scripts with to ensure their attacks, or defences, arrived within milliseconds of desired times.

This all was really far from the point of the game, which was a strategy. To remove this aspect of the game, in it's successor the developers added a random 0-30s variance to either side of the attack travel time. It worked pretty well, but people would still do things like sending attacks, checking if the time was close to optimal, and then cancelling and resending with quicker units if it was not.

Re: Overfitting and the strong version of Goodhart’s law

#100

Post by mannykannot »

AnthonyMouse wrote: post #96
mannykannot wrote: post #93

Earlier quoted context omitted.

Insurance is not free, and it generally gets more expensive the more reckless you are (or seem) to be (which is not to say this is the only factor in whether this is a zero-sum game, but it is arguably the most objectively quantifiable.)

The moral hazard is the difference between the insurance payout and the amount your future rates change as a result of the claim. If this amount is zero, there is no insurance. If it isn't, there is that much less incentive to avoid the harm.

Much less? Can you explain this immediate threshold as soon as the moral hazard is non-zero?
Post reply on HN