Overfitting and the strong version of Goodhart’s law
sohl-dickstein.github.io
Overfitting and the strong version of Goodhart’s law
1–10 of 111 posts
Re: Overfitting and the strong version of Goodhart’s law
#2Unnecessary disclosure:
I posted this article not only because it seems useful to me and others, but also because I'm not sure.
Edit: I'm not sure in which ways it wouldn't be.
Re: Overfitting and the strong version of Goodhart’s law
#3A system can be made efficient in a choice of different parameters; there is no one efficiency. The parameters which are not optimized can suffer. This is almost self-evident. Optimization makes plain what is valued, and sacrifices what is less valued.
Re: Overfitting and the strong version of Goodhart’s law
#4This keeps getting rediscovered in new domains.
JIT was the savior of manufacturing, until people learned that a single traffic jam that delayed a single delivery could create costs far in excess of the inventory savings.
Optimizations are critical, everywhere. And measuring optimizations is important because it is is easier and cheaper and earlier than measuring end results. Measuring days of inventory, or dollars in inventory, is a pretty good proxy for supply chain efficiency, until covid hits and suddenly “efficiency” means stability rather than minimum costs.
Over and over. Branch prediction / Spectre. Mortgage-Backed Securities. Optimizations based on second- and third-order effects blow up in the real world where any abstraction is approximate.
So it’s not the efficiency that’s the problem, it’s changing the focus from maximizing the desired output to maximizing efficiency, and measuring success based on the performance of efficiency optimizations.
Re: Overfitting and the strong version of Goodhart’s law
#5Everything is a tradeoff.
Efficiency is usually a tradeoff that limits flexibility. If you don't happen to need flexibility, you can have amazing efficiency: a wheel of a rail car is so much more efficient than a foot. But only when it stays on a rail.
If we look at a somehow more difficult terrain, a foot suddenly happens to be a better deal, because it adapts to a variety of conditions. Of course, at the expense of efficiency (complexity, energy consumption, need for a much more advanced control circuits, etc).
In business you usually have to have some safety margin for the unexpected. If you squeeze it out in the name of efficiency (and profit), all goes well until it does not, and then it fails much harder, maybe catastrophically.
(Nassim Taleb wrote a whole book about that, "Antifragility", where he gives a ton of fun real life examples to make basically the above point.)
Re: Overfitting and the strong version of Goodhart’s law
#6Product management 101; never choose a an objective metric without setting guardrail constraints.
Re: Overfitting and the strong version of Goodhart’s law
#7Product management 101; never choose a an objective metric without setting guardrail constraints.
Shouldn't it be: never choose an objective metric without choosing several other in conflict
Re: Overfitting and the strong version of Goodhart’s law
#8Everything is a tradeoff. Efficiency is usually a tradeoff that limits flexibility. If you don't happen to need flexibility, you can have amazing efficiency: a wheel of a rail car is so much more efficient than a foot. But only when it stays on a rail. If we look at a somehow more difficult terrain, a foot suddenly happens to be a better deal, because it adapts to a variety of conditions. Of course, at the expense of…
I would argue "efficiency" is the wrong word for what we discuss. Efficiency means optimising resource usage while achieving goals. If we need flexibility to achieve our goals consistently (and you usually do) then flexibility is part of efficiency (and effectiveness and efficacy) rather than opposed to it.
Phrased differently: if you define "efficiency" to mean "optimise for a single proxy metric and not what you're actually interested in" then yes, of course efficiency and effectiveness will be opposed other than sometimes by chance. But that's a dumb definition of efficiency!
Re: Overfitting and the strong version of Goodhart’s law
#9This keeps getting rediscovered in new domains. JIT was the savior of manufacturing, until people learned that a single traffic jam that delayed a single delivery could create costs far in excess of the inventory savings. Optimizations are critical, everywhere. And measuring optimizations is important because it is is easier and cheaper and earlier than measuring end results. Measuring days of inventory, or dollars i…
I feel like i have to repeat this very often: if a single traffic jam or other predictable common-cause variation results in your JIT implementation costing "far in excess of inventory savings" all you're telling me is that you have a really shitty JIT implementation.
JIT means having the buffer on hand to handle at the very least common-cause variation. But it also should come with the flexibility to handle variation of assignable causes at a reasonable cost.
Critically, you can't just press delete on all your inventory and then call it JIT. You have to adapt your processes for it, work with local authorities for improvements to infrastructure, and then always keep a larger buffer than you think you need.
JIT is not about deleting inventory. It's about reducing variation in your processes until most of your inventory truly becomes useless.
When you have a good JIT implementation, you are more flexible to take advantage of changing external conditions, not less.
Re: Overfitting and the strong version of Goodhart’s law
#10Great article. I have two comments:
1. Procrastination seems to be a type of early stopping. I knew I had a good strategy in school!
2. Something that seems to be sorely missing in machine learning (I'm not a ML expert) are error bars. If you take the example of the figure at the end, as you increase the number of parameters in the model, your error bars become larger (at least in the overfitting regime), and they are infinite when you have more parameters than data points. Indeed, chi^2 tests are usually used in physics/astro to test for this. Of course, you need error bars on the data points to do this. So perhaps the difficulty is really in assigning meaningful uncertainties to your pictures/test scores/politicians.