Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building high-level features using large scale unsupervised learning

research.google.com

Post reply on HN
Print view View original

101–110 of 196 posts

Re: Building high-level features using large scale unsupervised learning

#101

Post by Dn_Ab »

scotty79 wrote: post #66
shriphani wrote: post #61

Earlier quoted context omitted.

Babies can outperform any CV model with less data than that.

I strongly doubt that. I think babies have much richer input than any CV system up to date. In my opinion movement is crucial for understanding image.

A human baby learns in a very interesting way. By using an overcomplete basis to sparsely code data. This method has only recently started getting attention in ML.

A human baby learns from uncleaned raw data using far less energy with better generalization than a computer and fuses large amount of data without suffering from dimensionality curses.

I think it is safe to say that human babies are still ahead. for now.

Re: Building high-level features using large scale unsupervised learning

#102

Post by agravier »

abrahamsen wrote: post #78
agravier wrote: post #43

Earlier quoted context omitted.

> BUT the detector built its own categories(!). It's not revolutionary. Clustering algorithms and neural nets are plenty. Really, what differentiates this network is its scale.

And, thanks to scale, that some of the clusters corresponds to high level concepts. According to the article, earlier attempts have mostly resulted in low level concepts like "edge" or "blob" to be detected. Also, it was (again, from the article) plausible but not a given that high level concepts could be found from unlabeled data. That "cat" is one of the high level concept you get from using random Youtube videos a…

Exactly. Regarding your remark about edge detectors: such self-organizing neural nets are organized into hierarchical layers, and early layers' units are going to learn to become detectors of statistically common components of the input image, in the same way as the initial layers of the visual system perform blob and edge detection (retina, lateral geniculate nucleus, V1). In mathematical terms, these early units learn the conditional principal components of the inputs. The layers that are built upon these detectors, if correctly organized, are going to build upon this initial abstraction and learn more complex features: for instance to find these these edges in relative positions (to each other). Eventually, up the abstraction chain, units detect such statistically frequent features as the shape of cat's ears (common in youtube videos, I imagine), etc...

(Sorry I wrote that fast, I hope it's understandable)

Re: Building high-level features using large scale unsupervised learning

#103

Post by Dn_Ab »

ralfd wrote: post #83
p1esk wrote: post #50

Earlier quoted context omitted.

That's what I did a few months ago - quit my job and decided to go to a grad school to study AI (with focus on neural nets and ML).

What did you learn so far about neural nets? I recently looked into machine learning and naively thought I could find at least one practical fun tutorial "Here is a neural network API in C, you have to do that and this to let a simulated robot evade obstacles or learn to play Asteroids". Instead my (extremely superficial) search did find that neural nets are pretty arcane, genetic algorithms trapped in local minima a…

If you are trying to make game AI then neural networks are a bad idea. But if you are just trying to have some fun learning then do whatever you want. That said, MLPs are hard to use properly, not a good place to start.

A good way to start "AI". Write a decision tree they will serve you well and with boosting do even better. Basic but useful stuff: logistic regression, armed bandits, weighted experts, kNearest, k means , Kernel Density estimation and Naive Bayes. That covers online, ensemble, super and unsuper vised algorithms. Goodluck!

Re: Building high-level features using large scale unsupervised learning

#104

Post by nkoren »

JohnHaugeland wrote: post #91
marshallp wrote: post #12

Earlier quoted context omitted.

It does this for 20,000 different objects categories - this is getting close to matching human visual ability (and there are huge societal implications if computer reach that standard). This is the most powerful AI experiment yet conducted (publicly known).

"It does this for 20,000 different objects categories - this is getting close to matching human visual ability" No, it isn't. This classifier cannot identify theme variations, unknown rotations, will confuse new objects for objects it already knows, is unable to cope with camera distortion, needs fixed lighting, has no capacity for weather, does not work in the time you need to run away from a tiger, requires hundred…

I'm not a Machine Learning / AI expert, so I have to ask: if running a neural network on 16,000 cores with a training set of 10 million objects isn't cutting edge research -- and if running "far larger" networks than this, as you say you have, also isn't cutting edge research -- then please tell me: what is cutting edge research?

I ask this question in all seriousness; I'd really like to know.

(And yes, I see that your username is that of a noted AI researcher. Who died in 2010. So if you're actually his beta simulation, then I'll indeed be rather impressed...)

Re: Building high-level features using large scale unsupervised learning

#105

Post by [deleted] »

Cyranix wrote: post #13
marshallp wrote: post #2

Google Research and Stanford researchers test out a 1 billion connection, 9 layer, 16,000 core deep learning neural network (see geoff hinton and andrew ng talks on youtube) to recognize 20,000 different objects in images (with low accuracy but huge improvement over previous approaches). This was done no with no pre-labeled images (except for fine tuning)! A brain that learned from raw images. The same algorithm can…

"Singularity is near"? "Holy grail"? You may be getting a little carried away here. The outcome shows a very nice improvement on an unsupervised classification and feature detection task, but it also highlights that unsupervised machine learning still has a long way to go. 16% accuracy from a network with 1bn connections and 100m inputs using (if my math is right) 1.15m hours of CPU time. Which of these would be the…

[deleted]

Re: Building high-level features using large scale unsupervised learning

#106

Post by saalweachter »

JohnHaugeland wrote: post #89
reitzensteinm wrote: post #15

Earlier quoted context omitted.

Actually, what's significant about this work is that labeled training data was not required: "Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not."

Unsupervised training is not particularly significant, and was the original form of neural network. "our experimental results reveal" No, they don't. We had one of these in the 1980s.

Did we? A lot of early works in AI were ... /overstated/.[1] While a lot of concepts were created way back when, a lot of results weren't really generated. It's extremely valuable for someone to actually go and do a thing, now that we can, even if someone had the idea for the thing eons ago.

[1] http://dl.acm.org/citation.cfm?id=1045340

Re: Building high-level features using large scale unsupervised learning

#107

Post by Dn_Ab »

abrahamsen wrote: post #78
agravier wrote: post #43

Earlier quoted context omitted.

> BUT the detector built its own categories(!). It's not revolutionary. Clustering algorithms and neural nets are plenty. Really, what differentiates this network is its scale.

And, thanks to scale, that some of the clusters corresponds to high level concepts. According to the article, earlier attempts have mostly resulted in low level concepts like "edge" or "blob" to be detected. Also, it was (again, from the article) plausible but not a given that high level concepts could be found from unlabeled data. That "cat" is one of the high level concept you get from using random Youtube videos a…

The performance is due more to the architecture than to the scale. The scale is to handle all that data. But the feature learning performance has to do with their layered sparse learning technique which is brilliant. Although, their autoencoder neural network is actually learning a decomposition on the data so the neural part is kind of a red herring.

That you get high level features instead of edges is not the impressive part - you can just as well write a sparse non negative matrix factorization algorithm that will efficiently learn/represent eyes, lips and noses as features of faces unsupervised.

Re: Building high-level features using large scale unsupervised learning

#108

Post by caster_cp »

ralfd wrote: post #83
p1esk wrote: post #50

Earlier quoted context omitted.

That's what I did a few months ago - quit my job and decided to go to a grad school to study AI (with focus on neural nets and ML).

What did you learn so far about neural nets? I recently looked into machine learning and naively thought I could find at least one practical fun tutorial "Here is a neural network API in C, you have to do that and this to let a simulated robot evade obstacles or learn to play Asteroids". Instead my (extremely superficial) search did find that neural nets are pretty arcane, genetic algorithms trapped in local minima a…

SVMs are a relatively easy to use (but not to understand) method that yield impressive results for beginners. See the libsvm website, there are plenty of good material there (http://www.csie.ntu.edu.tw/~cjlin/libsvm/). But as stated somewhere else here, game AI is a whole different story.

Re: Building high-level features using large scale unsupervised learning

#109

Post by Symmetry »

seiji wrote: post #76

16,000 cores sounds impressive until you realize it's just five to ten modern GPUs. For Google, it's easier to just run a 1,000 machine job than requisition some GPUs. See: http://www.nvidia.com/object/tesla-servers.html (4.5 teraflops in one card ) Reminder: GPUs will destroy the world.

What a GPU calls a "core" doesn't at all correspond to what a CPU calls a "core". Going by the CPU definition (something like "something that can schedule memory accesses") a high end GPU will only have 60 or so cores. And going by the the GPU definition (An execution unit) a high end CPU will tend to have 30-something cores.

GPUs do have fundamentally more execution resources, but that comes at a price and not every algorithm will be capable of running faster on a GPU than on a CPU. If neural networks just involve multiplying lots of matrices together with little branching they might be well suited to GPUs, but most AI code isn't like that.

Post reply on HN