Machine learning has become alchemy (2017) [video]
1–10 of 134 posts
Re: Machine learning has become alchemy (2017) [video]
#2Science is a combination of theory and experiment. Sometimes theory advances faster than experiment, sometimes vice versa. Right now in ML, experiment aka practice is advancing faster than theory. Theory will eventually catch up.
Re: Machine learning has become alchemy (2017) [video]
#3It is simple to say deep learning is based on "alchemy" or "engineering" or whatever it is that isn't strong theory. And it's reasonable to say deep learning has a lot of mathematical and statistical intuitions but doesn't have a strong theory - maybe just doesn't yet have a strong theory or maybe can never get one.
So this is by now a standard argument. The standard answers I think have been:
1) Well, we are discovering that you don't need a strong theory for a truth-discovery machine. We've discovered experimental thinking construction.
2) It's true deep learning doesn't have a strong theory now but sooner or later it will get one.
3) This shows that instead we need theory X, usually by people who've been pursuing theory X all along. But I think Hinton at least has thrown a bunch of alternatives against the wall over the years.
(I could swear this has appeared here before but I can't find it).
Re: Machine learning has become alchemy (2017) [video]
#4Science is a combination of theory and experiment. Sometimes theory advances faster than experiment, sometimes vice versa. Right now in ML, experiment aka practice is advancing faster than theory. Theory will eventually catch up.
one would hope so!
the depressing thing about machine learning to me is the many convincing explanations of phenomena, that then turn out not to explain things (all the explanations of why dropout is effective, or why there are adversarial examples, things like that.)
this is where the comparison to alchemy is the sharpest in my opinion. The alchemists had extremely sophisticated theories that they used to provide explanations for the phenomena they observed; just ultimately none of them made any sense.
Re: Machine learning has become alchemy (2017) [video]
#5[deleted]
Re: Machine learning has become alchemy (2017) [video]
#6Science is a combination of theory and experiment. Sometimes theory advances faster than experiment, sometimes vice versa. Right now in ML, experiment aka practice is advancing faster than theory. Theory will eventually catch up.
one would hope so! the depressing thing about machine learning to me is the many convincing explanations of phenomena, that then turn out not to explain things (all the explanations of why dropout is effective, or why there are adversarial examples, things like that.) this is where the comparison to alchemy is the sharpest in my opinion. The alchemists had extremely sophisticated theories that they used to provide ex…
I think people end up reaching for straws because a lot of this stuff works well in practice and they’re trying to make sense of it. 20 years from now we’ll have a much clearer picture of why the techniques work! Until then, a lot of the explanations, maybe even some valid ones, may sound like baloney!
Re: Machine learning has become alchemy (2017) [video]
#7[deleted]
Re: Machine learning has become alchemy (2017) [video]
#8Science is a combination of theory and experiment. Sometimes theory advances faster than experiment, sometimes vice versa. Right now in ML, experiment aka practice is advancing faster than theory. Theory will eventually catch up.
ML is a souped-up version of "draw a line through these points". We can get really efficient at drawing lines through points, but it's not like we'll suddenly realize some deeper fundamental theory about it!
Re: Machine learning has become alchemy (2017) [video]
#9Science is a combination of theory and experiment. Sometimes theory advances faster than experiment, sometimes vice versa. Right now in ML, experiment aka practice is advancing faster than theory. Theory will eventually catch up.
ML is a souped-up version of "draw a line through these points". We can get really efficient at drawing lines through points, but it's not like we'll suddenly realize some deeper fundamental theory about it!
The reason you don't expect to see a deep fundamental theory of drawing a line through a few points is because you can always do it. ML doesn't always work, and sometimes it is harder to get working than other times. What's going on?
Re: Machine learning has become alchemy (2017) [video]
#10Can someone give a concrete example of the kind of theoretical properties they desire of ``new-style'' machine learning? The kinds of properties that ``old-style'' learning methods guaranteed?
People often complain about interpretability: in what sense is an SVM interpretable that a deep neural network is not?
Or is the worry about gradient descent not finding global optima? But why is the global optimum a satisfactory place to be, if the theory does not also provide a satisfactory connection between the space of models and underlying reality?
The arbiter of good theory is ultimately its ability to guide and explain practical phenomena. Which machine learning phenomena are currently most in need of theoretical elucidation?