Do you know what the scaling law actually is? Overfit everything as much as you can.
Why don't machine learning research agents overfit?
71–77 of 77 posts
Re: Why don't machine learning research agents overfit?
#72No point in reading anything AI related anymore. It's all slop. We need to retvrn to rss feeds
Is there a study demonstrating that AI-assisted articles are less substantive or is it purely a kneejerk bias towards humanslop?
Re: Why don't machine learning research agents overfit?
#73If anything, the latest generation of AI models, Astra and Fable, are prime example of overfitting—whereas benchmarks suggest they’re AGI-tier, users (including myself) report the same old gaslighting, hallucination, context rot, cheating, incomprehensibility patterns as with prior models, sometimes even more pronounced. Fable and Opus 5, I suspect, will become textbook examples of RL collapse.
[dead]
Re: Why don't machine learning research agents overfit?
#74Earlier quoted context omitted.
This is a longstanding principle in model-fitting. More parameters, almost always, improves the ability of the model to fit to any particular data, in-sample. The model with the least parameters is both the simplest in principle and has the best chance of not overfitting.
This is provably not true, and you can use the marginal likelihood / PAC-Bayes to prove it (or any other framework for measuring model quality). Increase the number of parameters in a linear model way beyond the point of interpolation, and concentrate the likelihood around the zero loss set. Then reduce the variance on a Gaussian prior. You can balance the two temperatures at exactly the right rate so that any measur…
You are saying something interesting, but talking like Grok and skipping a lot of the details, without any references to common check-in points like terminology or specific studies.
> and concentrate the likelihood around the zero loss set. Then reduce the variance on a Gaussian prior.
Those phrases could mean a lot of different things. What are you proposing?
> so that any measure of model quality will monotonically increase with model size and achieve a maximum at infinite model size.
any measure of model quality? You must have some bounds of any measure, since trivially that's false because "fewer parameters is better" is a measure of model quality, even if dumb.
It's hard to even engage when you're being so imprecise, and not even giving one specific example.
Re: Why don't machine learning research agents overfit?
#75Even tech giants are putting out articles seemingly fully written by Claude.
Time to first detected slop in this article is Is it too much to ask from people to read their own article anymore? If anyone read this at all, they would have had the ick, and would have fired off a prompt to get rid of the most popular AI slop tells...
It is even wrong about things
> A validation set is one you consult repeatedly while building the model — to compare candidates, tune hyperparameters, and decide what to try next.
Re: Why don't machine learning research agents overfit?
#76Even tech giants are putting out articles seemingly fully written by Claude.
Time to first detected slop in this article is Is it too much to ask from people to read their own article anymore? If anyone read this at all, they would have had the ick, and would have fired off a prompt to get rid of the most popular AI slop tells...
"The more your listener already knows, the shorter the message you need to send. An expert ML engineer needs only a few sentences; a newcomer needs the whole manual."
Literally from the first words on the page. 1st sentence passed barely. 2nd sentence ending "a newcomer needs the whole manual" and it's Claude.
Re: Why don't machine learning research agents overfit?
#77Earlier quoted context omitted.
I think you should get less annoyed. > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct https://web.archive.org/web/20170428225156/http://www.cse.bu... https://web.archive.org/web/20130412062821/http://cs.ecs.bay...
There are also various metaphysical theories that posit that the universe is algorithmically generated in some sense or the other, and from many of those theories it follows that simplicity is a fundamental feature of reality, which yields an even stronger version of Occam’s Razor.
You don't need the onerous assumption that the universe is algorithmically generated, just that all ways to reason about and communicate intelligently for the purpose of making decisions is digital.