Earlier quoted context omitted.
The animated graphic labeled "Occam's razor, formalized" is bizarre. Is that really visualizing "Occam's razor, formalized"?
I was expecting Occam wearing a suit.
Animated SVG of a pelican shaving with Occam's razor?
Why don't machine learning research agents overfit?
61–70 of 70 posts
Re: Why don't machine learning research agents overfit?
#62Earlier quoted context omitted.
And I like the bayesian interpretation too. Murphy's "Probabilistic machine learning" has an occam's razor section. The idea is that a complex model explains many more configurations (datasets) than a simple one. So its (prior) probability distribution is lower on the data seen (to compensate for the other possibilities it might explain). So the (marginal) likelihood that the simple model is correct is higher if it f…
This is a longstanding principle in model-fitting. More parameters, almost always, improves the ability of the model to fit to any particular data, in-sample. The model with the least parameters is both the simplest in principle and has the best chance of not overfitting.
This is provably not true, and you can use the marginal likelihood / PAC-Bayes to prove it (or any other framework for measuring model quality). Increase the number of parameters in a linear model way beyond the point of interpolation, and concentrate the likelihood around the zero loss set. Then reduce the variance on a Gaussian prior. You can balance the two temperatures at exactly the right rate so that any measure of model quality will monotonically increase with model size and achieve a maximum at infinite model size.
Even easier, just take a limit of polynomial regression to a Gaussian process while optimizing the marginal likelihood over the prior temperature.
In all of these cases, the model with the least parameters is not the simplest in principle and does not have the best chance of not overfitting. The reality is significantly more nuanced.
Re: Why don't machine learning research agents overfit?
#63Earlier quoted context omitted.
I think you should get less annoyed. > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct https://web.archive.org/web/20170428225156/http://www.cse.bu... https://web.archive.org/web/20130412062821/http://cs.ecs.bay...
Another useful link [0], page 91. Effectively, the more complex the solution, the heavier is the upper bound on true risk. It doesn't mean a simpler model is necessarily better. But the complexity brings its own larger support for mistakes to live in. The _likelihood_ of being _more correct_ is probably related to larger sample required to learn a more complex model. [0]: https://www.cs.huji.ac.il/~shais/Understandin…
Absolutely not. This link is a reference on PAC learning, which is thoroughly misleading in the land of deep learning and inevitably leads to vacuous bounds. This is common knowledge in deep learning. I would not recommend that any student learn any part of this theory at this point, since we have far better alternatives in terms of simplicity, accuracy, and generality.
PAC-Bayes is genuinely superior, for example. Instead of a uniform weighting over all hypotheses (effectively encoded in the supremum), you get to weight the hypothesis class in virtually any way that you want. This is critical to ensure that you exclude absurd hypotheses that you have almost nil chance of reaching. If you do consider a uniform weighting, then you can just easily reduce to PAC anyway, but you do so in a cleaner package.
Re: Why don't machine learning research agents overfit?
#64Earlier quoted context omitted.
The notion of "simplicity" can be completely arbitrary, though. It's enough that there are only finitely many hypotheses simpler than the best hypothesis (assuming there's such a thing as a best hypothesis). So as you eliminate hypotheses incompatible with the data, at some point you'll have eliminated all simpler hypotheses, and the simplest hypothesis left will also be the best hypothesis. If simpler hypotheses are…
Except for the fact that eventually we are all dead. So it is kind of important to get there faster. For complicated hypotheses, where complicated is defined appropriately, it takes many many examples to realize that it was a wrong hypothesis all along. There lies the rub. For a particular instance of a learning problem we can't tell much, however using a Occams razor over many instances, one would be correct more of…
[deleted]
Re: Why don't machine learning research agents overfit?
#65[flagged]
Re: Why don't machine learning research agents overfit?
#66Earlier quoted context omitted.
Time to first detected slop in this article is Is it too much to ask from people to read their own article anymore? If anyone read this at all, they would have had the ick, and would have fired off a prompt to get rid of the most popular AI slop tells...
What I really dislike is having to edit my own non-LLM assisted writing to make sure I'm not accidentally confused with AI. I caught myself writing "And that matters because..." in a HN comment but had to edit myself. Also miss uising emdashes.
For the first time ever in my life, I consider the fact that English is not my first language and that I never had any formal education in it as an advantage.
Nobody is going to consider my weird and barbarian prose with the well polished product of a SOTA model.
Re: Why don't machine learning research agents overfit?
#67Earlier quoted context omitted.
What I really dislike is having to edit my own non-LLM assisted writing to make sure I'm not accidentally confused with AI. I caught myself writing "And that matters because..." in a HN comment but had to edit myself. Also miss uising emdashes.
For the first time ever in my life, I consider the fact that English is not my first language and that I never had any formal education in it as an advantage. Nobody is going to consider my weird and barbarian prose with the well polished product of a SOTA model.
English as a 2nd language speakers often have better grammar than natives in my experience! I wouldn't be surprised if the average native english speaker doesn't know the different between there and their.
Re: Why don't machine learning research agents overfit?
#68I always get annoyed when people misinterpret Occam’s razor. It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. It’s just like the Hopper quote. She said it’s better to ask for forgiveness during the fog of war, doing something you thought was right, not to do something you knew they were going to say no to and now you are trying to get away with something.
“You should prefer it, because it’s simple” is just restating Occam’s razor, not giving any explanation. “The simplest is more likely to be accurate” is a much better interpretation than yours.
I think the most accessible example of Occam’s razor is fitting a line to some points; you can always use a high enough order polynomial to fit the seen points exactly, but a straight line is likely closer to representing the underlying distribution.
Re: Why don't machine learning research agents overfit?
#69Earlier quoted context omitted.
For the first time ever in my life, I consider the fact that English is not my first language and that I never had any formal education in it as an advantage. Nobody is going to consider my weird and barbarian prose with the well polished product of a SOTA model.
English as a 2nd language speakers often have better grammar than natives in my experience! I wouldn't be surprised if the average native english speaker doesn't know the different between there and their.
Their, their, mate. You mispelled they're as there in your comment they're
Re: Why don't machine learning research agents overfit?
#70Earlier quoted context omitted.
For the first time ever in my life, I consider the fact that English is not my first language and that I never had any formal education in it as an advantage. Nobody is going to consider my weird and barbarian prose with the well polished product of a SOTA model.
English as a 2nd language speakers often have better grammar than natives in my experience! I wouldn't be surprised if the average native english speaker doesn't know the different between there and their.
If you saw what goes on in middle school classrooms of native English speakers, I’d think you’d be in for a surprise…