Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Overfitting and the strong version of Goodhart’s law

sohl-dickstein.github.io

Post reply on HN
Print view View original

21–30 of 111 posts

Re: Overfitting and the strong version of Goodhart’s law

#21

Post by jaschasd »

a9h74j wrote: post #20

Is there any application of Goodhart's law to competing in elections? "Getting a winning number of votes" seems to be one kind of metric. How might a system adapt?

(blog post author here)

Yes! The post actually talks about that a bit: how overfitting can result from treating "leaders that have the most support in the population" as a proxy for "leaders that act in the best interests of the population"; and some ideas for improving that with noise regularization or capacity restrictions.

Re: Overfitting and the strong version of Goodhart’s law

#22

Post by astrobe_ »

brookst wrote: post #4

This keeps getting rediscovered in new domains. JIT was the savior of manufacturing, until people learned that a single traffic jam that delayed a single delivery could create costs far in excess of the inventory savings. Optimizations are critical, everywhere. And measuring optimizations is important because it is is easier and cheaper and earlier than measuring end results. Measuring days of inventory, or dollars i…

> This keeps getting rediscovered in new domains.

I'm not sure there's a need for a "strong version" of Goodhart's law, or I fail to understand the distinction the author is trying to make.

Goodhart is "so true" precisely because it warns about the fact that the measure is often not the goal itself, just like "the map is not the territory", and is indeed an approximation of the goal.

One finds a similar problem with the word "best": best according to which metric? That's how people have different ideas of what is the best.

Re: Overfitting and the strong version of Goodhart’s law

#23

Post by visarga »

epgui wrote: post #11

I think there are some similarities between Goodhart’s law and overfitting, but I don’t think the lesson or underlying mechanism or model is the same. Crudely speaking, Goodhart’s law is a reflection of the system reacting to an intervention (ie.: dynamic feedback loops) and that just has nothing to do with overfitting.

Training a neural net is a dynamic feedback loop too. Back-propagation is the feedback phase.

Re: Overfitting and the strong version of Goodhart’s law

#24

Post by kqr »

phkx wrote: post #15
kqr wrote: post #8

Earlier quoted context omitted.

I would argue "efficiency" is the wrong word for what we discuss. Efficiency means optimising resource usage while achieving goals. If we need flexibility to achieve our goals consistently (and you usually do) then flexibility is part of efficiency (and effectiveness and efficacy) rather than opposed to it. Phrased differently: if you define "efficiency" to mean "optimise for a single proxy metric and not what you're…

To me, efficiency means achieving a desired goal while consuming a minimum of resources (time, energy, space, …). If the goal is defined in a too narrow scope, i.e. your ‚dumb‘ definition of efficiency, the flexibility may be missing. Still, that particular goal may be reached efficiently. So it’s not an issue with the definition of efficiency, but rather with scoping the problem. As the article states, it may not al…

It seems to me you're making the same terminology mistake, only shifted one layer back! You're expressing yourself as if the "too narrow metrics" were the goal in and of themselves. I would argue the metrics are never the goal -- the fuzzy intention they're proxying for is always the goal.

Re: Overfitting and the strong version of Goodhart’s law

#25

Post by cbrogrammer »

> Goal: Distribution of labor and resources based upon the needs of society > Proxy: Capitalism > Strong version of Goodhart's law leads to: Massive wealth disparities (with incomes ranging from hundreds of dollars per year to hundreds of dollars per second), with more than a billion people living in poverty

Nothing as annoying as online communists inserting their horrible political opinions everywhere. Try moving to a communist country. See how capitalism has raied billions out of poverty in only 30 years

Re: Overfitting and the strong version of Goodhart’s law

#26

Post by salawat »

kqr wrote: post #9
brookst wrote: post #4

This keeps getting rediscovered in new domains. JIT was the savior of manufacturing, until people learned that a single traffic jam that delayed a single delivery could create costs far in excess of the inventory savings. Optimizations are critical, everywhere. And measuring optimizations is important because it is is easier and cheaper and earlier than measuring end results. Measuring days of inventory, or dollars i…

I feel like i have to repeat this very often: if a single traffic jam or other predictable common-cause variation results in your JIT implementation costing "far in excess of inventory savings" all you're telling me is that you have a really shitty JIT implementation. JIT means having the buffer on hand to handle at the very least common-cause variation. But it also should come with the flexibility to handle variatio…

You've now redefined JIT to mean "buffered with the minimum viable buffer" where minimum viable buffer is "whereever I drop the goalposts when defining common-cause variation.

It's the definitional version of working out a variable in an equation, making a mistake, and ending up looking at 0=0.

Happens all the time.

Re: Overfitting and the strong version of Goodhart’s law

#27

Post by ToJans »

This is exactly what Taleb wrote about [1].

Systems start out very fragile. Over time they get more and more robust, as edge cases and events result in more rules and rigidity. This works until they become so robust that they can no longer cope with the change when something unexpected -i.e. a black swan event - happens.

The only way to counter this, to make a system that improves in coping with disorder/chaos as it encounters it.

[1] https://en.wikipedia.org/wiki/Antifragility

Update: reworded that last sentence a bit.

Re: Overfitting and the strong version of Goodhart’s law

#28

Post by ShamelessC »

whack wrote: post #13

I'm going to go on a rant here. Most times when I hear someone invoke Goodhart's law, it's because they are opposed to transparent and metric-driven decision-making. It's easy to find the flaws with this approach, as the article has effectively done. But what they inevitably ignore is the flaws of not using this approach. Without a transparent metrics-driven decision making process, what you end up with one of the fo…

The article isn't a critique of teams following Goodhart's Law IRL. It's a formalization of concepts between machine learning and Goodhart's Law including a "strong" formalization where performance can diverge.

I found it quite useful to think about these concepts with respect to machine learning. In a sufficiently large system of humans interacting, there's a lot of emergent stochasticity that is tough to model.

Some examples given are of teams working together at a business, but seemed intentionally written to strike familiarity with the reader, not to make suggestions about how people should analyze metrics on a daily basis.

Re: Overfitting and the strong version of Goodhart’s law

#29

Post by kqr »

salawat wrote: post #26
kqr wrote: post #9

Earlier quoted context omitted.

I feel like i have to repeat this very often: if a single traffic jam or other predictable common-cause variation results in your JIT implementation costing "far in excess of inventory savings" all you're telling me is that you have a really shitty JIT implementation. JIT means having the buffer on hand to handle at the very least common-cause variation. But it also should come with the flexibility to handle variatio…

You've now redefined JIT to mean "buffered with the minimum viable buffer" where minimum viable buffer is "whereever I drop the goalposts when defining common-cause variation. It's the definitional version of working out a variable in an equation, making a mistake, and ending up looking at 0=0. Happens all the time.

Yes, but that it literally what JIT means, though. It's a mindset of solving assignable-cause variation at its cause rather than papering over it with more inventory.

Re: Overfitting and the strong version of Goodhart’s law

#30

Post by visarga »

JuettnerDistrib wrote: post #10

Great article. I have two comments: 1. Procrastination seems to be a type of early stopping. I knew I had a good strategy in school! 2. Something that seems to be sorely missing in machine learning (I'm not a ML expert) are error bars. If you take the example of the figure at the end, as you increase the number of parameters in the model, your error bars become larger (at least in the overfitting regime), and they ar…

> as you increase the number of parameters in the model, your error bars become larger

In large neural nets the effect is reversed. The larger the model, the better it generalises, even from the same training data.

Post reply on HN