Even tech giants are putting out articles seemingly fully written by Claude.
[dead]
Why don't machine learning research agents overfit?
31–40 of 71 posts
Re: Why don't machine learning research agents overfit?
#32No point in reading anything AI related anymore. It's all slop. We need to retvrn to rss feeds
what would returning to rss feeds achieve?
Re: Why don't machine learning research agents overfit?
#33I always get annoyed when people misinterpret Occam’s razor. It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. It’s just like the Hopper quote. She said it’s better to ask for forgiveness during the fog of war, doing something you thought was right, not to do something you knew they were going to say no to and now you are trying to get away with something.
I think you should get less annoyed. > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct https://web.archive.org/web/20170428225156/http://www.cse.bu... https://web.archive.org/web/20130412062821/http://cs.ecs.bay...
None of your links work for me.
Re: Why don't machine learning research agents overfit?
#34Earlier quoted context omitted.
I think you should get less annoyed. > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct https://web.archive.org/web/20170428225156/http://www.cse.bu... https://web.archive.org/web/20130412062821/http://cs.ecs.bay...
None of your links work for me.
Ah! from my very dated and messy bibtex file comments. Wait, let me search for them on archive.org.
Fixed.
Re: Why don't machine learning research agents overfit?
#35Compression in this modern day and age is so slop.
Yes, I'm familiar with keystone results such as Solomonoff induction. It's a direct counterexample to compression - your intensional algorithm can completely outrun reality. I can literally specify a huge mega-algorithm that just searches over all possible Turing machines and evaluates them, and it's an optimal compressor. It's completely vacuous though. You can always hide the "heavy work" in your mappings and descriptions. It's ironic that a kolomogorov complexity minimizer is so loaded that it's vacuous.
This is pretty much why I roll my eyes at this point at all the compression is intelligence memes.
I wonder when intervention and causality will hit the mainstream. These tools were designed specifically to counteract purely predictive theories. But your average compression dude will hold tight to their paradigms and slogans, not realize their internal contradictions (that their own field has brought up), and then whenever a new paradigm suddenly becomes visible and mainstream, they'll latch onto that. It's not principled at all.
And to be clear - I do think intelligence is some amount of compression, and I am well aware of formal results such as the arithmetic decoding theoretical and empricial result. Just annoyed. It's literally no different than the whole Bayesianism meme. If you're not actually practicing that type of intelligence as a basis, then you don't get to go around beating the drum about how it's the ultimate reality. You're just spouting dogma to feel like part of an in-group.
Re: Why don't machine learning research agents overfit?
#36I always get annoyed when people misinterpret Occam’s razor. It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. It’s just like the Hopper quote. She said it’s better to ask for forgiveness during the fog of war, doing something you thought was right, not to do something you knew they were going to say no to and now you are trying to get away with something.
Marcus Hutter formalized this in his AIXI work.
Re: Why don't machine learning research agents overfit?
#37I always get annoyed when people misinterpret Occam’s razor. It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. It’s just like the Hopper quote. She said it’s better to ask for forgiveness during the fog of war, doing something you thought was right, not to do something you knew they were going to say no to and now you are trying to get away with something.
And sadly, in academia, complexity (opposite of Occam's razor) is what gets you published.
It's partly because of disagreement about what "complexity" and "simplicity" actually are. Many simple statements are in fact backed by massively complex, unstated assumptions. In attempting to deal with assumptions, scientists necessarily end up having to deal with the complexity involved in that. But the problem is if you don't engage with that, how do you know what is really more conformant with Occam's razor, as opposed to just satisfying what is readily expressed in common language?
Likewise in software development a C developer will say abstractions are not simple, a java programmer will argue that dealing with low levels details is not simple. They're both kinda right but will resort to framings which back their world view.
Re: Why don't machine learning research agents overfit?
#38Re: Why don't machine learning research agents overfit?
#39I always get annoyed when people misinterpret Occam’s razor. It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. It’s just like the Hopper quote. She said it’s better to ask for forgiveness during the fog of war, doing something you thought was right, not to do something you knew they were going to say no to and now you are trying to get away with something.
I think you should get less annoyed. > It’s not that the simplest is more likely to be correct, it’s that you should prefer it, because it’s simple. I don't know what Occam meant, but if you accept the formalism of PAC learning, it is more likely to be correct https://web.archive.org/web/20170428225156/http://www.cse.bu... https://web.archive.org/web/20130412062821/http://cs.ecs.bay...
The notion of "simplicity" can be completely arbitrary, though. It's enough that there are only finitely many hypotheses simpler than the best hypothesis (assuming there's such a thing as a best hypothesis). So as you eliminate hypotheses incompatible with the data, at some point you'll have eliminated all simpler hypotheses, and the simplest hypothesis left will also be the best hypothesis. If simpler hypotheses are also more likely to be correct, you get there faster, but it's not required.
Re: Why don't machine learning research agents overfit?
#40> Machine learning, at its core, is about generalization, not memorization. Well they memorize the patterns. memorization doesnt mean rote learning.
Does '110001111000000011111111111' contain n>1∧∀d(d|n→(d=1∨d=n)) as well as infinite number of other generalizations?