Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building high-level features using large scale unsupervised learning

research.google.com

Post reply on HN
Print view View original

171–180 of 196 posts

Re: Building high-level features using large scale unsupervised learning

#171

Post by karpathy »

If anyone is interested to read more on this topic, there is another recent, closely related and perhaps slightly more accessible paper ( "High-Level Invariant Features with Scalable Clustering Algorithms" http://bit.ly/KDuN04 ) from Stanford that also learns face neurons from unsupervised collection of images (disclaimer: I'm co-author). It uses a slightly different model based on layers of k-means clustering and linking, but the computation in the end is very similar.

I'm familiar with both models so I can also try to answer any questions.

Re: Building high-level features using large scale unsupervised learning

#172

Post by mturmon »

JohnHaugeland wrote: post #163
mturmon wrote: post #147

Earlier quoted context omitted.

I'm not a large scale ML person, and not intending to take away from the achievement of the team in the OP, but experiments in large scale, unsupervised learning have been going for a long time (even using the autoencoder approach). When you think about it, large scale requires unsupervised... Here is an old example with hundreds of millions of records and instances: http://aaaipress.org/Papers/KDD/1998/KDD98-028.pdf…

"When you think about it, large scale requires unsupervised..." Not necessarily. Crowdsourcing is another option, like Google's image tagging game, reCAPTCHA, et cetera. Pay a herd of people to do things, and they'll do things for you. You don't have to pay them in money. Telling them they have a high score is often enough.

Yes, "requires" was too strong. I should have said they go well together. I was trying to get at the fact that it's highly common for large-scale work to be unsupervised.

Re: Building high-level features using large scale unsupervised learning

#173

Post by mturmon »

marshallp wrote: post #161
mturmon wrote: post #147

Earlier quoted context omitted.

I'm not a large scale ML person, and not intending to take away from the achievement of the team in the OP, but experiments in large scale, unsupervised learning have been going for a long time (even using the autoencoder approach). When you think about it, large scale requires unsupervised... Here is an old example with hundreds of millions of records and instances: http://aaaipress.org/Papers/KDD/1998/KDD98-028.pdf…

Face recognition usually uses a hand-coded layer followed by a machine learning algorithm. This technique automatically devises that hand-coded layer. It also did this for 20,000 other categories and can also be applied equally well to audio or any other data type. Huge difference.

The large-number-of-categories result was the most novel and surprising to me.

Re: Building high-level features using large scale unsupervised learning

#174

Post by moultano »

mbq wrote: post #79

As far as I can tell, this is let's train a huge number of models and then cherry-pick few that works well on a test set, so an overfitted junk. What have I missed?

It seems like they are cherry picking the one that works well on the training set no? Where did you get the sense that they were doing it on test?

Re: Building high-level features using large scale unsupervised learning

#175

Post by Dn_Ab »

JohnHaugeland wrote: post #165
MaysonL wrote: post #157

Earlier quoted context omitted.

In general, a singularity is a point at which an equation, surface, etc., blows up or becomes degenerate. Singularities are often also called singular points. To call Vinge a particularly bad science fiction author says more about your critical acumen than about him. (Perhaps you're thinking about his ex-wife?) (I read the "singularity is near" in the article title as ironic - almost parodic).

"In general, a singularity is a point at which an equation, surface, etc., blows up or becomes degenerate." Do you also participate in discussions of Microsoft Surface by explaining that in general, a surface is a flat exterior of a coherent object? . "To call Vinge a particularly bad science fiction author" Vinge didn't come up the singularity; Kurtzweil did. Kurtzweil did. I quite like Vernor Vinge's work. . "says…

Vinge did coin the term singularity.

The term was coined by science fiction writer Vernor Vinge, who argues that artificial intelligence, human biological enhancement or brain-computer interfaces could be possible causes of the singularity.

http://en.wikipedia.org/wiki/Technological_singularity

Re: Building high-level features using large scale unsupervised learning

#176

Post by snikolov »

agravier wrote: post #43
nl wrote: post #33

People are missing the point here. Yes, 15% accuracy doesn't seem great. BUT the detector built its own categories(!). It managed to find 20,000 different categories of objects in Youtube videos, and one of these categories corresponded to human faces, and another to cats. Once the experimenters found the "face detection neuron" and used it to test faces THAT neuron managed 81.7% detection rate(!). Forget the singula…

> BUT the detector built its own categories(!). It's not revolutionary. Clustering algorithms and neural nets are plenty. Really, what differentiates this network is its scale.

Clustering algorithms operate on features, which typically have to be designed by hand. The appeal of deep learning is that it discovers good features automatically.

Re: Building high-level features using large scale unsupervised learning

#178

Post by heretohelp »

roel_v wrote: post #56
heretohelp wrote: post #18

Earlier quoted context omitted.

Haha, no. --- last company was in computer vision.

"last company was in computer vision." Did you sell, leave or did it fail? Why? I have some ideas that I think are novel applications of computer vision, and just within the range of what's feasible, but it seems that most computer vision applications look like that at first, and then after 90% done find out that the second 90% is exponentially harder and, realistically, infeasible. How could I test my ideas against…

Second reply:

Email me if you want to discuss practical application.

Re: Building high-level features using large scale unsupervised learning

#179

Post by MaysonL »

Dn_Ab wrote: post #175
JohnHaugeland wrote: post #165

Earlier quoted context omitted.

"In general, a singularity is a point at which an equation, surface, etc., blows up or becomes degenerate." Do you also participate in discussions of Microsoft Surface by explaining that in general, a surface is a flat exterior of a coherent object? . "To call Vinge a particularly bad science fiction author" Vinge didn't come up the singularity; Kurtzweil did. Kurtzweil did. I quite like Vernor Vinge's work. . "says…

Vinge did coin the term singularity. The term was coined by science fiction writer Vernor Vinge, who argues that artificial intelligence, human biological enhancement or brain-computer interfaces could be possible causes of the singularity. http://en.wikipedia.org/wiki/Technological_singularity

His essay is available at:

http://www-rohan.sdsu.edu/faculty/vinge/misc/singularity.htm...

Re: Building high-level features using large scale unsupervised learning

#180

Post by MaysonL »

JohnHaugeland wrote: post #120
saalweachter wrote: post #106

Earlier quoted context omitted.

Did we? A lot of early works in AI were ... /overstated/.[1] While a lot of concepts were created way back when, a lot of results weren't really generated. It's extremely valuable for someone to actually go and do a thing, now that we can, even if someone had the idea for the thing eons ago. [1] http://dl.acm.org/citation.cfm?id=1045340

The early works in AI with regards to unsupervised learning were in the 1940s and 1950s. Claude Shannon had demonstrated a chess learning system which taught itself by playing him to defeat him in under two weeks as early as 1949. No, they weren't overstated. They were hyped by a clueless press. There's a pretty critical difference. It's a bit like how the early web pioneers didn't say that the web was going to revol…

Claude Shannon had demonstrated a chess learning system which taught itself by playing him to defeat him in under two weeks as early as 1949.

I call "citation needed", and needed quite badly: Shannon was a much better chess player than any program available in 1949.

Post reply on HN