Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Overfitting and the strong version of Goodhart’s law

sohl-dickstein.github.io

Post reply on HN
Print view View original

71–80 of 111 posts

Re: Overfitting and the strong version of Goodhart’s law

#71

Post by jaschasd »

kkoncevicius wrote: post #32
jaschasd wrote: post #17

Earlier quoted context omitted.

Author of blog post here! Overfitting can happen in many ways -- your training objective can be different at train and test time, or as you suggest the datapoints you use can be different at train and test time. For overfitting induced by datapoints: If you include the datapoints in your problem specification, then you can say they induce bias at test time. If you treat the choice of training datapoints as a random v…

My first impression is also in agreement with the parent. The blog post appears to use some terms loosely in order to make the connection between overfitting and Goodhart's law stronger. For example - calling training sample "proxy" and stating that it is is a slightly different goal is already leading towards the pre-defined conclusion. And the reply also leaves me with a similar impression: > your training objectiv…

A gentle note that an incomplete piece of a goal (e.g. a loss function computed on a subset of the data) is a proxy for the full goal (e.g. the loss function on the full dataset).

Similarly, concept drift can be a source of overfitting -- the objective you care about is the one after the concept drift occurred, but the objective you trained on is the one from before the concept drift. (Here's a scholar search for papers where the two concepts co-occur: https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&as_vis... )

Re: Overfitting and the strong version of Goodhart’s law

#72

Post by jaschasd »

bigfudge wrote: post #44

A lot of the suggested mitigations made a lot of sense to me. For example reducing time to prepare grant applications, or to add 1s jitter to stock trades. Perhaps there are flaws to these specific ideas, but the thing that struck me most was that we have almost no way of implementing ideas like this in Anglo political countries right now. Entrenched interests, and the technical detail involved makes it hard to imagi…

+1 to this comment! The barrier between (thinking we) know what changes should happen, and realizing those changes in the real world, is complex and frustrating and political.

Re: Overfitting and the strong version of Goodhart’s law

#73

Post by cs702 »

Except that AI models, especially large deep ones, do NOT overfit like the author thinks. They exhibit what is now called "deep double descent" -- the validation error declines, then increases, and then declines again:

https://openai.com/blog/deep-double-descent/

A question I've pondered for a while is whether complex systems in the real world also exhibit double descent.

For example, transitioning an online application that currently serves thousands of users to one that can serve millions and then billions requires reorganizing all code, processes, and infrastructure, making software development harder at first, but easier down the road. Anyone who has gone through it will tell you that it's like going through "phase transitions" that require "getting over humps."

Similarly, startups that want to transition from small-scale to mid-size and then to large-scale businesses must increase operational complexity, making everything harder at first, but easier down the road. Anyone who has been with a startup that has grown from tiny two-person shop to large corporation will tell you that it's like going through "phase transitions" that require "getting over humps."

Finally, it may be that whole countries and economies that want to improve the lives of their citizens may have to go through an interim period of less efficiency, making everything harder at first, but easier down the road. It may be that human progress involves "phase transitions" that require "getting over humps."

Re: Overfitting and the strong version of Goodhart’s law

#74

Post by jaschasd »

alexmolas wrote: post #50

Nice article! Just a weird question, which theme are you using? I liked it, so I tried to check the source code in your GitHub, because your are using a `.github.io` domain I assumed you were using GitHub Pages to host the blog, however I didn't find it in your repo, which is really weird.

Yup, it's GitHub pages. It's just a private repo, so the world doesn't get to see my embarrassing edits and half written drafts. I would be happy to share a snapshot of the source code with you if you have a specific use for it though -- email me.

I'm using the minima Jekyll theme. I also used Markdeep (https://casual-effects.com/markdeep/) rather than vanilla markdown to write the post. (Markdeep is awesome. It took me a full day to figure out how to get it to play nice with Jekyll, but in the end of course it turned out to be simple. You can see how I did it by looking at the blog page source -- note the block, and the "mode: 'html'" option.)

Re: Overfitting and the strong version of Goodhart’s law

#75

Post by jaschasd »

langsoul-com wrote: post #37

The strong version of Goodharts law seems occur as a function of time. In the beginning, everything is fine and dandy, but as people optimise, it begins to turn into extremes.

+1. It doesn't require there to be a time axis -- but in practice, we almost always optimize incrementally, so it takes a while for the strong version of Goodhart's law to kick in.

Re: Overfitting and the strong version of Goodhart’s law

#77

Post by jaschasd »

TomSwirly wrote: post #39
jaschasd wrote: post #19

Earlier quoted context omitted.

Blog post author here. Just for the record, I am very much in favor of metric-driven decision making. I suggest some ways we can make it more robust, and also that we should be aware that our metrics may not be measuring what we intend.

This was a great article - I'm sending it to a couple of my friends. Might I ask what software you used for blogging? I can't seem to find the source repo it came from...

See response to alexmolas -- I'm using GitHub Pages + Jekyll + Markdeep. You don't see the source repo because it's private, but I'm happy to share a code snapshot with you if you like -- email me for it.

Re: Overfitting and the strong version of Goodhart’s law

#78

Post by jaschasd »

cs702 wrote: post #73

Except that AI models, especially large deep ones, do NOT overfit like the author thinks. They exhibit what is now called "deep double descent" -- the validation error declines, then increases, and then declines again : https://openai.com/blog/deep-double-descent/ A question I've pondered for a while is whether complex systems in the real world also exhibit double descent. For example, transitioning an online applica…

Blog post author here.

A brief note that I do discuss the deep double descent phenomenon in the blog. See the section starting with "One of the best understood causes of extreme overfitting is that the expressivity of the model being trained too closely matches the complexity of the proxy task."

I avoided using the actual term double descent, since I thought it would add unnecessary complexity. Lesson learned for next time -- I should have at least had an endnote using that terminology!

Re: Overfitting and the strong version of Goodhart’s law

#79

Post by jaschasd »

lqet wrote: post #33

I recommend reading "The Collapse of Complex Societies" by Joseph Tainter [0]. Complex societies tend to address problems by adding more and more rules and regulations, simply because they have always done so and it has been successful in the past. More importantly, though, it is typically the only tool they have. Essentially these societies are increasingly overfitting their legislature to narrow special cases until…

Thanks! I'm also adding this to my reading list now.

Re: Overfitting and the strong version of Goodhart’s law

#80

Post by AnthonyMouse »

mannykannot wrote: post #64
bombcar wrote: post #62

Earlier quoted context omitted.

Something like this could (should?) perhaps be handled by governmental "remedy" service, a small insurance/tax against everyone that the judges can use to award remedy without forcing the defendant to pay.

We have already moved on, and a case like this would no longer be regarded as unusual, let alone difficult. In the process, something like what you propose arose, in the form of liability insurance. The system works reasonably well, in that there are a lot fewer hard cases without creating a crippling burden in the average case. If anything, this case, together with the way things have changed since then, demonstrate…

But that only moves the problem somewhere else, because insurance is a moral hazard. Someone who would otherwise be cautious to prevent harm has less incentive to do it because when the harm comes the insurance pays.
Post reply on HN