Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Hinton’s Google Move Says About the Future of Machine Learning

moalquraishi.wordpress.com

Post reply on HN
Print view View original

31–36 of 36 posts

Re: What Hinton’s Google Move Says About the Future of Machine Learning

#31

Post by ilaksh »

davmre wrote: post #14
ilaksh wrote: post #4

I think that ml people should take a look at the AGI field. I also think that more powerful techniques, specialized hardware like qualcomms baby Brain corporation are building, and/or large peer computing networks will make general intelligence accessible for small groups or individuals In fewer than twenty years.

AGI has cool ideas, and is in some sense the "right" theoretical framework for AI, but it's not clear that it gives any kind of practical path forward for AI research. The main problem is that its basic idea -- an AI performing Bayesian inference over a hypothesis class of all potential environment-generating computer programs, with a Kolmogorov complexity prior -- is wildly uncomputable, so to make it practical we'd…

"an AI performing Bayesian inference over a hypothesis class of all potential environment-generating computer programs, with a Kolmogorov complexity prior, -- is wildly uncomputable, so to make it practical we'd need to find simple, computable approximations that work on real problems"

That's not what AGI is trying to do or how they are trying to do it.

Re: What Hinton’s Google Move Says About the Future of Machine Learning

#32

Post by wookietrader »

ilaksh wrote: post #31
davmre wrote: post #14

Earlier quoted context omitted.

AGI has cool ideas, and is in some sense the "right" theoretical framework for AI, but it's not clear that it gives any kind of practical path forward for AI research. The main problem is that its basic idea -- an AI performing Bayesian inference over a hypothesis class of all potential environment-generating computer programs, with a Kolmogorov complexity prior -- is wildly uncomputable, so to make it practical we'd…

"an AI performing Bayesian inference over a hypothesis class of all potential environment-generating computer programs, with a Kolmogorov complexity prior, -- is wildly uncomputable, so to make it practical we'd need to find simple, computable approximations that work on real problems" That's not what AGI is trying to do or how they are trying to do it.

It's at least one way which has been advocated by leading researcher of the field. If you think differently, you should give references and explain what your AGI definition is.

Re: What Hinton’s Google Move Says About the Future of Machine Learning

#34

Post by aspis »

davmre wrote: post #11

As an ML researcher, this article isn't persuasive to me for a few reasons: - Computing power is getting exponentially cheaper even as computing requirements increase. The resources available to a university lab in the future will be much greater than those available today, even given the same budget. Of course this is also true for industry, but this growth is not a unique advantage of industry. - Other scientific f…

I attended a talk by Quoc Le at UCSD recently, and he made the case that it is necessary to get the algorithms tested large scale, rather than sending too much time on it at small scale.

He had presented a graph comparing some models and their accuracy as the number of features was scaled up to the tens of thousands, his point being that some models that work best at smaller number of features fall off as the number is scaled up. Unfortunately the slides he has on his web page is outdated, so I haven't been able to find that reference. I'd be very happy if one of you know which paper he was referring to. In the old slides he refers to this paper, which makes something of the same point: http://ai.stanford.edu/~ang/papers/nipsdlufl10-AnalysisSingl... It shows how simple unsupervised models with dense feature extraction reach the state of the art performance of more complex models.

Of course, I can see how it makes sense to at least do some small scale prototyping, to work out kinks like you say - but the lesson is that if you are planning to do large scale machine learning you can't necessarily use the small scale tests as a good guide for large scale performance. It's certainly promising if you get very good accuracy, speed or both at small scale, though neither necessarily will carry over to large scale. On the flip side, if your method is worse than state-of-the-art at smaller scales, that doesn't mean it won't beat state-of-the-art at large scales.

Re: What Hinton’s Google Move Says About the Future of Machine Learning

#36

Post by jmares »

aspis wrote: post #34
davmre wrote: post #11

As an ML researcher, this article isn't persuasive to me for a few reasons: - Computing power is getting exponentially cheaper even as computing requirements increase. The resources available to a university lab in the future will be much greater than those available today, even given the same budget. Of course this is also true for industry, but this growth is not a unique advantage of industry. - Other scientific f…

I attended a talk by Quoc Le at UCSD recently, and he made the case that it is necessary to get the algorithms tested large scale, rather than sending too much time on it at small scale. He had presented a graph comparing some models and their accuracy as the number of features was scaled up to the tens of thousands, his point being that some models that work best at smaller number of features fall off as the number…

Data shows, as you say, that small scale performance is no indicator of large scale performance.

How then do you decide which projects are worth trying on the large scale?

Post reply on HN