Why don't machine learning research agents overfit?

Post by hodgehog11 » Wed, Sep 16, 2026, 12:22 AM UTC

jmalicki wrote: post #98

hodgehog11 wrote: post #95

Earlier quoted context omitted.

No, I am talking about out of sample error and estimates thereof. It is "overfitting" to data, but it also has lower out of sample error than the case where you do not "overfit". This is why the notion of overfitting is not nearly as cut and dry as a basic ML course would have you believe. Just because you fit data exactly does not mean that your estimator has high error on out of sample data. A trivial counterexampl…

Nothing in your reply gets at the connection to out of sample data?

I don't understand what you mean. Test error is literally out of sample error. Marginal likelihood is designed to estimate out of sample error. The whole discussion is about out of sample; nothing has been about in-sample error. The in-sample error for my examples are all trivially zero, so only out of sample error is worth discussing.

Post by vrighter » Wed, Sep 16, 2026, 12:09 PM UTC

1718627440 wrote: post #93

vrighter wrote: post #85

Earlier quoted context omitted.

simpler is not the right word either. it's the one that makes the least assumptions, not the simplest. The simplest would be "god did it" pretty much everytime.

> The simplest would be "god did it" pretty much everytime. An out-of universe entity, that is by definition too large to be understandable to anything in universe, is a lot, but not simple. Are you sure, you are not confusing easy and simple?

It is simple. There aren't any explanations needed because "god is omnipotent" already covers everything.

But its simplicity involves making MASSIVE assumptions.

And the fact that Occam's razor explicitly says "fewest assumptions" and not "simple" is unchanged.

Quantum mechanics is complicated. Assuming electrons orbit nuclei is simple. But quantum mechanics is backed by countless experiments, so needs less assumptions. If you have to choose between quantum mechanics' description of the atom, or the old, outdated, knowledge of an atom, Occam's Razor would say to choose quantum mechanics.

Re: Why don't machine learning research agents overfit?

Re: Why don't machine learning research agents overfit?