Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why don't machine learning research agents overfit?

amazon.science

Post reply on HN
Print view View original

41–50 of 71 posts

Re: Why don't machine learning research agents overfit?

#41

Post by zmgsabst »

yorwba wrote: post #39
srean wrote: post #23

Earlier quoted context omitted.

I think you should get less annoyed. > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct https://web.archive.org/web/20170428225156/http://www.cse.bu... https://web.archive.org/web/20130412062821/http://cs.ecs.bay...

The notion of "simplicity" can be completely arbitrary, though. It's enough that there are only finitely many hypotheses simpler than the best hypothesis (assuming there's such a thing as a best hypothesis). So as you eliminate hypotheses incompatible with the data, at some point you'll have eliminated all simpler hypotheses, and the simplest hypothesis left will also be the best hypothesis. If simpler hypotheses are…

You choice of basis matters, eg, wavelet versus sinusoid.

Re: Why don't machine learning research agents overfit?

#42

Post by srean »

yorwba wrote: post #39
srean wrote: post #23

Earlier quoted context omitted.

I think you should get less annoyed. > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct https://web.archive.org/web/20170428225156/http://www.cse.bu... https://web.archive.org/web/20130412062821/http://cs.ecs.bay...

The notion of "simplicity" can be completely arbitrary, though. It's enough that there are only finitely many hypotheses simpler than the best hypothesis (assuming there's such a thing as a best hypothesis). So as you eliminate hypotheses incompatible with the data, at some point you'll have eliminated all simpler hypotheses, and the simplest hypothesis left will also be the best hypothesis. If simpler hypotheses are…

Except for the fact that eventually we are all dead. So it is kind of important to get there faster.

For complicated hypotheses, where complicated is defined appropriately, it takes many many examples to realize that it was a wrong hypothesis all along. There lies the rub.

For a particular instance of a learning problem we can't tell much, however using a Occams razor over many instances, one would be correct more often than not. Provided, of course, the PAC assumptions are true or they are not very far from being true.

How far is not very far ? That gets very hairy to quantify.

Re: Why don't machine learning research agents overfit?

#43

Post by miltava »

srean wrote: post #23
diddid wrote: post #8

I always get annoyed when people misinterpret Occam’s razor. It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. It’s just like the Hopper quote. She said it’s better to ask for forgiveness during the fog of war, doing something you thought was right, not to do something you knew they were going to say no to and now you are trying to get away with something.

I think you should get less annoyed. > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct https://web.archive.org/web/20170428225156/http://www.cse.bu... https://web.archive.org/web/20130412062821/http://cs.ecs.bay...

And I like the bayesian interpretation too. Murphy's "Probabilistic machine learning" has an occam's razor section.

The idea is that a complex model explains many more configurations (datasets) than a simple one. So its (prior) probability distribution is lower on the data seen (to compensate for the other possibilities it might explain). So the (marginal) likelihood that the simple model is correct is higher if it fits the data well enough.

[1] https://probml.github.io/pml-book/book1.html

Re: Why don't machine learning research agents overfit?

#44

Post by [deleted] »

srean wrote: post #42
yorwba wrote: post #39

Earlier quoted context omitted.

The notion of "simplicity" can be completely arbitrary, though. It's enough that there are only finitely many hypotheses simpler than the best hypothesis (assuming there's such a thing as a best hypothesis). So as you eliminate hypotheses incompatible with the data, at some point you'll have eliminated all simpler hypotheses, and the simplest hypothesis left will also be the best hypothesis. If simpler hypotheses are…

Except for the fact that eventually we are all dead. So it is kind of important to get there faster. For complicated hypotheses, where complicated is defined appropriately, it takes many many examples to realize that it was a wrong hypothesis all along. There lies the rub. For a particular instance of a learning problem we can't tell much, however using a Occams razor over many instances, one would be correct more of…

[deleted]

Re: Why don't machine learning research agents overfit?

#45

Post by srean »

miltava wrote: post #43
srean wrote: post #23

Earlier quoted context omitted.

I think you should get less annoyed. > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct https://web.archive.org/web/20170428225156/http://www.cse.bu... https://web.archive.org/web/20130412062821/http://cs.ecs.bay...

And I like the bayesian interpretation too. Murphy's "Probabilistic machine learning" has an occam's razor section. The idea is that a complex model explains many more configurations (datasets) than a simple one. So its (prior) probability distribution is lower on the data seen (to compensate for the other possibilities it might explain). So the (marginal) likelihood that the simple model is correct is higher if it f…

True.

It so happens that one gets the best generalization error bounds when one combines PAC with Bayesian ideas -- the PAC-Bayesian bounds.

Re: Why don't machine learning research agents overfit?

#46

Post by srean »

sigbottle wrote: post #35

Compression in this modern day and age is so slop. Yes, I'm familiar with keystone results such as Solomonoff induction. It's a direct counterexample to compression - your intensional algorithm can completely outrun reality. I can literally specify a huge mega-algorithm that just searches over all possible Turing machines and evaluates them, and it's an optimal compressor. It's completely vacuous though. You can alwa…

> I can literally specify a huge mega-algorithm that just searches over all possible Turing machines and evaluates them

Then why don't you ? and did you mean all or those that halt ? I presume you have a way of separating those.

Re: Why don't machine learning research agents overfit?

#47

Post by jsrozner »

Why is this being published as a blog post and not as a peer-reviewed submission? If it's going to be a blog post, why isn't there a corresponding scientific version for me to look at?

Someone else already found it. I don't understand why the link isn't in the blog post. https://arxiv.org/abs/2606.11045

Use of claude for writing it should be disclosed.

Re: Why don't machine learning research agents overfit?

#48

Post by [deleted] »

jsrozner wrote: post #47

Why is this being published as a blog post and not as a peer-reviewed submission? If it's going to be a blog post, why isn't there a corresponding scientific version for me to look at? Someone else already found it. I don't understand why the link isn't in the blog post. https://arxiv.org/abs/2606.11045 Use of claude for writing it should be disclosed.

[deleted]

Re: Why don't machine learning research agents overfit?

#49

Post by gwerbin »

miltava wrote: post #43
srean wrote: post #23

Earlier quoted context omitted.

I think you should get less annoyed. > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct https://web.archive.org/web/20170428225156/http://www.cse.bu... https://web.archive.org/web/20130412062821/http://cs.ecs.bay...

And I like the bayesian interpretation too. Murphy's "Probabilistic machine learning" has an occam's razor section. The idea is that a complex model explains many more configurations (datasets) than a simple one. So its (prior) probability distribution is lower on the data seen (to compensate for the other possibilities it might explain). So the (marginal) likelihood that the simple model is correct is higher if it f…

This is a longstanding principle in model-fitting. More parameters, almost always, improves the ability of the model to fit to any particular data, in-sample. The model with the least parameters is both the simplest in principle and has the best chance of not overfitting.

Re: Why don't machine learning research agents overfit?

#50

Post by tomrod »

yorwba wrote: post #39
srean wrote: post #23

Earlier quoted context omitted.

I think you should get less annoyed. > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct https://web.archive.org/web/20170428225156/http://www.cse.bu... https://web.archive.org/web/20130412062821/http://cs.ecs.bay...

The notion of "simplicity" can be completely arbitrary, though. It's enough that there are only finitely many hypotheses simpler than the best hypothesis (assuming there's such a thing as a best hypothesis). So as you eliminate hypotheses incompatible with the data, at some point you'll have eliminated all simpler hypotheses, and the simplest hypothesis left will also be the best hypothesis. If simpler hypotheses are…

There is some art to it, especially in model architecture choices.

But your smarter data scientists will try to get the best information density on test data to approximate generalization. MDL, as championed here, or AIC/BIC if working with more established and acceptable methods.

Post reply on HN