Building high-level features using large scale unsupervised learning • Hacker News Board

Re: Building high-level features using large scale unsupervised learning

#31

Post by leot » Fri, Jun 22, 2012, 6:30 AM UTC

Smerity wrote: post #17

jpeterson wrote: post #5

Maybe I'm missing something here, but how exactly is it "unlabeled" data if they're specifically feeding it millions of pictures of "faces"? I mean, if you make a specific selection of the type of images you train the network on, isn't that basically equivalent to labeling them?

The aim of the paper was to produce an unsupervised system that would generate high level features from noisy data. These high level features could then be used in supervised systems where labelled data is added. Thus, the paper is about using an unsupervised system to help a later supervised system. An advantage of this is that, as the unsupervised system isn't trained to recognise object X, it instead learns featur…

In other words: imagine a baby. She sees 100k "images" of faces. Thanks to the statistical regularity of the world, she now has a "subsystem" that recognizes a face in the absence of her knowing what it's called. Then, when she is told "this is a face" she pins this thing pointed-to to the existing, unnamed representation.

Re: Building high-level features using large scale unsupervised learning

#32

Post by marshallp » Fri, Jun 22, 2012, 6:31 AM UTC

forgotusername wrote: post #6

Can we perhaps edit "singularity is near" out of the title? This sounds impressive, but having a bunch of racks able to classify the outline of a face is vastly disconnected from machine and humanity merging.

I put the singularity bit in to make it relevant for those who are non-technical. This experiment is significant because it shows that large artificial neural networks can be made to work. People have tried and failed at this for decades.

This technigue was "discovered" by geoff hinton at the university of toronto in 2005. However, nobody at tried (or maybe got enough funds) to try it this scale.

If this continues to work at larger and larger scale, this would be a machine learning technique that can work accurately on tasks that are hugely important to society

- accurate speech recognition - human level compuer vision (make human manual labor redundant)

Re: Building high-level features using large scale unsupervised learning

#33

Post by nl » Fri, Jun 22, 2012, 6:32 AM UTC

People are missing the point here.

Yes, 15% accuracy doesn't seem great.

BUT the detector built its own categories(!). It managed to find 20,000 different categories of objects in Youtube videos, and one of these categories corresponded to human faces, and another to cats.

Once the experimenters found the "face detection neuron" and used it to test faces THAT neuron managed 81.7% detection rate(!).

Forget the singularity, and just think about how amazing that is. The system trained itself - without human labelling - to distinguish human faces correctly over 80% of the time.

Re: Building high-level features using large scale unsupervised learning

#34

Post by nl » Fri, Jun 22, 2012, 6:32 AM UTC

kenrikm wrote: post #7

While I'm excited about progress, 15.8% accuracy is not exactly "Singularity is near"

It detects faces with 81% accuracy. That isn't bad at all...

Also, the previous best on the same dataset was 9.3%

Re: Building high-level features using large scale unsupervised learning

#35

Post by panacea » Fri, Jun 22, 2012, 6:50 AM UTC

MindTwister wrote: post #30

panacea wrote: post #27

The singularity is already here. There are black holes in our universe. This submission title really annoys me.

Even if you are trolling, I think I'll leave this here. http://en.wikipedia.org/wiki/Technological_singularity

I'm not trolling, I've read Keurzweil's book... It's extrapolating. If I was a risk assesor, I'd err on the side of a mass extinction event over us consuming the entire universe as data.

Re: Building high-level features using large scale unsupervised learning

#36

Post by scotty79 » Fri, Jun 22, 2012, 6:51 AM UTC

ekianjo wrote: post #25

15.8% accuracy != singularity is near. No more supplements-eating Kurzweil, walking Terminators and Skynet-like BS please.

I wonder what accuracy would human get if you trained him/her only with 10 million static 200x200 px images in complete silence.

Re: Building high-level features using large scale unsupervised learning

#37

Post by Smerity » Fri, Jun 22, 2012, 6:55 AM UTC

reitzensteinm wrote: post #15

Smerity wrote: post #10

Earlier quoted context omitted.

I was going to make the same request. The singularity should be discussed where relevant, not added to everything. This paper is producing high level features from noisy data in an unsupervised fashion -- a human still needs to indicate the task it should be targeted for and a human still needs to provide labelled training data for these high level features to be of use. This work is interesting enough to warrant det…

Actually, what's significant about this work is that labeled training data was not required: "Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not."

I replied by adding to my comment above as it wouldn't allow me to reply earlier. Reference that.

tldr; High-level features generated by this unsupervised network are so high-level that one of them aligns with "has a face in the image", another to "has cat in image" (see the section "Cat and human body detectors") and so on. Note however that they select the "best neuron" for face classification -- the only way they can do that is via using labelled data and testing all the neurons (where each neuron's activation is a feature). Thus, these features cannot be used without labelled training.

Re: Building high-level features using large scale unsupervised learning

#38

Post by ekianjo » Fri, Jun 22, 2012, 6:57 AM UTC

scotty79 wrote: post #36

ekianjo wrote: post #25

15.8% accuracy != singularity is near. No more supplements-eating Kurzweil, walking Terminators and Skynet-like BS please.

I wonder what accuracy would human get if you trained him/her only with 10 million static 200x200 px images in complete silence.

I don't think this was "complete silence" in this case. They actually trained the computer (meaning they provided data as input).

Re: Building high-level features using large scale unsupervised learning

#39

Post by zhouyisu » Fri, Jun 22, 2012, 7:12 AM UTC

marshallp wrote: post #8

chubot wrote: post #4

Earlier quoted context omitted.

I only read the abstract, so I'm sure this is a basic/dumb question... but if you don't label images as faces or not, what makes it a face detector? :) How do you get an elbow detector or a butt detector out of the same algorithm?

It automatically creates a set features that you can then use a final layer of machine learning to get what you want. In machine learning, normally you have to create a set of features (call feature engineering - basically think algorithms to better represent your data). The amazing thing about deep learning is that the computer does this for you! You just need a few 10s/100s face/nonface images - same for 20,000 oth…

This paper is actually more interesting: it automatically learns some "neuron" which its firing represents a detected face, without any supervise technique. It shows the possibility to extract complex information solely from data.

Re: Building high-level features using large scale unsupervised learning

#40

Post by robryan » Fri, Jun 22, 2012, 7:14 AM UTC

tansey wrote: post #28

marshallp wrote: post #12

Earlier quoted context omitted.

It does this for 20,000 different objects categories - this is getting close to matching human visual ability (and there are huge societal implications if computer reach that standard). This is the most powerful AI experiment yet conducted (publicly known).

> It does this for 20,000 different objects categories With 15.8% accuracy. > This is the most powerful AI experiment yet conducted (publicly known). It's only powerful because they threw more cores at it than anyone else has previously attempted. From a quick skimming of the paper, there does not appear to be a lot of novel algorithmic contribution here. It's the same basic autoencoder that Hinton proposed years ago…

That in itself is fairly interesting, it says we can make dramatic improvements just throwing more processing power at the problem. Whatever happens on the algorithms research side of the problem in coming years, you can count on us having access to more processing power.