Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building high-level features using large scale unsupervised learning

research.google.com

Post reply on HN
Print view View original

121–130 of 196 posts

Re: Building high-level features using large scale unsupervised learning

#121

Post by campnic »

JohnHaugeland wrote: post #113
nkoren wrote: post #104

Earlier quoted context omitted.

I'm not a Machine Learning / AI expert, so I have to ask: if running a neural network on 16,000 cores with a training set of 10 million objects isn't cutting edge research -- and if running "far larger" networks than this, as you say you have, also isn't cutting edge research -- then please tell me: what is cutting edge research? I ask this question in all seriousness; I'd really like to know. (And yes, I see that yo…

I'm his son. Let's take the example of The Netflix Prize, a $1 million bounty that the movie shipping organization ran several years ago. Their purpose was to improve their ratings prediction algorithm, under the pretext that people frequently ran out of ideas of what to rent, and that a successful suggestion algorithm would keep people as customers longer after that point. So, they carefully defined the success rate…

I appreciate your insight into the original article and your help placing it into the context of the broader field.

Re: Building high-level features using large scale unsupervised learning

#122

Post by magoghm »

Devilboy wrote: post #52
sown wrote: post #41

I'm seriously considering quitting my job and studying ML for a few months in a desperate attempt to get work in projects like this. I feel like I'm missing out but too dumb for traditional grad school.

Have you seen the online courses? https://www.coursera.org/course/ml (From one of the authors of this paper!) https://www.coursera.org/course/vision https://www.coursera.org/course/computervision Prof. Hinton's videos are very watchable: http://www.youtube.com/watch?v=AyzOUbkUf3M http://www.youtube.com/watch?v=VdIURAu1-aU

If you like math, Caltech's "Learning from Data" is awesome http://work.caltech.edu/telecourse.html

Re: Building high-level features using large scale unsupervised learning

#123

Post by Dn_Ab »

scotty79 wrote: post #114
Dn_Ab wrote: post #101

Earlier quoted context omitted.

A human baby learns in a very interesting way. By using an overcomplete basis to sparsely code data. This method has only recently started getting attention in ML. A human baby learns from uncleaned raw data using far less energy with better generalization than a computer and fuses large amount of data without suffering from dimensionality curses. I think it is safe to say that human babies are still ahead. for now.

I agree that CV is currently behind human sight. I just argue that this is more due to lacking input processing than to actual classification engine. The reason they used 200x200 px images, I think, was that bigger images couldn't be analysed in sufficient quantity. In a way their first input processing of the real images was pixelisation filter. If you feed pixelised image to a person you see how much information is…

>>To improve CV we should focus on finding best ways of converting full resolution visual data to something of smaller volume in such way that important features are preserved.

You are exactly right! See Dictionary Learning, Random projection, compressive sensing. As for time, perhaps you are right I don't know. That question is: would a suitably written video trained classifier that preserved temporal features do better on image classification?

Re: Building high-level features using large scale unsupervised learning

#124

Post by Symmetry »

JohnHaugeland wrote: post #118
Symmetry wrote: post #109

Earlier quoted context omitted.

What a GPU calls a "core" doesn't at all correspond to what a CPU calls a "core". Going by the CPU definition (something like "something that can schedule memory accesses") a high end GPU will only have 60 or so cores. And going by the the GPU definition (An execution unit) a high end CPU will tend to have 30-something cores. GPUs do have fundamentally more execution resources, but that comes at a price and not every…

"What a GPU calls a "core" doesn't at all correspond to what a CPU calls a "core"." They aren't as different as you imagine. They're general purpose programmable arithmetic units with processing rates on the order of 20-30% of CPUs, provided the limitation that they're all doing roughly the same thing. For most machine learning tasks, that's exactly what you're doing anyway. Oh no, your neural network engine has to b…

A CPU core isn't a general purpose programmable arithmetic unit, though. In fact what you call a "core" when you're talking about CPUs is composed of multiple such general purpose programmable, as well as less general purpose memory load/store units that can still be used for basic arithmetic and a instruction fetch and scheduling system. So a core in you Intel iFOO processor is structurally equivalent to what NVidia calls an SM. Now, and NVidia SM has 48 execution units to Intels 6, but it operates at a lower frequency and doesn't have the bypass network, branch predictor, memory prefetcher, etc that you could find in an Intel core. So there are some tasks where the Intel core will be much faster than the NVidia SM, and some tasks where the NVidia SM will be much faster. And the case here does seem like one where the GPU has an advantage. But saying that the NVidia GPU has 1526 "cores" is just dishonest.

Re: Building high-level features using large scale unsupervised learning

#125

Post by robrenaud »

JohnHaugeland wrote: post #113
nkoren wrote: post #104

Earlier quoted context omitted.

I'm not a Machine Learning / AI expert, so I have to ask: if running a neural network on 16,000 cores with a training set of 10 million objects isn't cutting edge research -- and if running "far larger" networks than this, as you say you have, also isn't cutting edge research -- then please tell me: what is cutting edge research? I ask this question in all seriousness; I'd really like to know. (And yes, I see that yo…

I'm his son. Let's take the example of The Netflix Prize, a $1 million bounty that the movie shipping organization ran several years ago. Their purpose was to improve their ratings prediction algorithm, under the pretext that people frequently ran out of ideas of what to rent, and that a successful suggestion algorithm would keep people as customers longer after that point. So, they carefully defined the success rate…

An image is a lot more complicated than a pair of ids and a rating. Counting the number of rows in the training database is misleading. I can build a reasonable dataset for a prediction task from a set of 100M rows from a database that I maintain in my spare time (http://councilroom.com , predict player actions given partial game states).

Don't get me wrong, the Netflix prize was cool.

What's cool about this is that Google hasn't given the learning system a high level task. They basically say, figure out a lossy compression for these 10 million images. And then when they examine that compression method, they find that it can effectively generate human faces and cats.

Re: Building high-level features using large scale unsupervised learning

#126

Post by Dn_Ab »

Symmetry wrote: post #116
JohnHaugeland wrote: post #87

The singularity is a poorly constructed myth. It is built around the presumption that intelligence is a linear function of CPU power, and that surely as CPU power rises, so shall intelligence; the problem is, that prediction was made in the 1970s, since which CPU power has risen ten decimal orders of magnitude, and we still don't have much better speech recognition than we did back then, let alone anything even appro…

Well, maybe. There are a whole lot of very different things called "The Singularity" and some of them are much more reasonable than others. There's the Cambpellian Singularity, which says that we won't be able to predict what will happen next. Pretty non-controversial as far as it goes. There's the Vingean Singularity, which says that if we ever develop AIs that can think as fast and as well as humans then due to Moo…

Your characterization of Vinge's singularity is incorrect. I have never read anything in which he brings up infinity, I.J. Good does though. Vinge's is actually more like your AI revolution except that it will be evident as a singularity only to those looking forward and not to those looking backwards. So instead of using agriculture/industrial divide as an analogy he posits a human/animal divide.

As his definition of singularity is pretty strongly tied to comprehension think of it like this - the singularity is the time point after which a 10 year old unmodified human child from 1000 AD can not grow up to understand his or her surroundings.

Re: Building high-level features using large scale unsupervised learning

#127

Post by redmoskito »

JohnHaugeland wrote: post #120
saalweachter wrote: post #106

Earlier quoted context omitted.

Did we? A lot of early works in AI were ... /overstated/.[1] While a lot of concepts were created way back when, a lot of results weren't really generated. It's extremely valuable for someone to actually go and do a thing, now that we can, even if someone had the idea for the thing eons ago. [1] http://dl.acm.org/citation.cfm?id=1045340

The early works in AI with regards to unsupervised learning were in the 1940s and 1950s. Claude Shannon had demonstrated a chess learning system which taught itself by playing him to defeat him in under two weeks as early as 1949. No, they weren't overstated. They were hyped by a clueless press. There's a pretty critical difference. It's a bit like how the early web pioneers didn't say that the web was going to revol…

I find both of your comments extremely condescending, both toward saalweachter and the authors of this article.

1. The fact that Claude Shannon succeeded in training a chess system has virtually no impact on sallweachter's claim that many AI results were overstated.

2. Certainly the press overstated them, which supports saalweachter's premise rather than weakening it. Even if the _implied_ claim was that _researchers_ overstated results, your argument does nothing to weaken this claim.

3. Frito Lay solved a problem several orders of magnitude easier that of face recognition in natural images, which is still very much an open problem in computer vision.

4. Similar to 1., the Frito Lay example contributes nothing to your goal of weakening saalweachter's claim that this is valuable research--a claim which is exceedingly innocuous.

I understand that you've probably got a bone to pick against the many AI naysayers and saalweachter's comments conjured a few common misrepresentations (i.e. (a) that the "AI revolution" burnt-out because it's researchers were somehow naive and (b) that neural networks are something new invented by computer vision researchers). You'd be justified in arguing against these claims, and I'm sure your father (respected AI researcher of the same name) would make them too, if saalweachter had tried to make them (which he didn't). But even if you were justified in making the argument, I would expect a less condescending one that made better use of evidence than the argument you've made here.

Re: Building high-level features using large scale unsupervised learning

#128

Post by robrenaud »

JohnHaugeland wrote: post #92
marshallp wrote: post #23

Earlier quoted context omitted.

That's true. Let's say in machine learning then.

Let's not, because it's still wrong on the order of thirty counterexamples. Let's say we'll stop making broad proclamations about the global best in a field we know very little about.

Andrew Ng has set many state of the art results on various data sets using similar approaches as the one described in the paper.

Here is a reasonably approachable talk he gave about it. http://www.youtube.com/watch?v=ZmNOAtZIgIk

Re: Building high-level features using large scale unsupervised learning

#130

Post by JohnHaugeland »

Symmetry wrote: post #124
JohnHaugeland wrote: post #118

Earlier quoted context omitted.

"What a GPU calls a "core" doesn't at all correspond to what a CPU calls a "core"." They aren't as different as you imagine. They're general purpose programmable arithmetic units with processing rates on the order of 20-30% of CPUs, provided the limitation that they're all doing roughly the same thing. For most machine learning tasks, that's exactly what you're doing anyway. Oh no, your neural network engine has to b…

A CPU core isn't a general purpose programmable arithmetic unit, though. In fact what you call a "core" when you're talking about CPUs is composed of multiple such general purpose programmable, as well as less general purpose memory load/store units that can still be used for basic arithmetic and a instruction fetch and scheduling system. So a core in you Intel iFOO processor is structurally equivalent to what NVidia…

"In fact what you call a "core" when you're talking about CPUs is composed of multiple such general purpose programmable, as well as less general purpose memory load/store units"

So are GPU cores.

"But saying that the NVidia GPU has 1526 "cores" is just dishonest."

No, it isn't. You can run 1536 things in parallel at speeds that would have qualified as full cpu speeds several years prior.

Something isn't any less a core merely because it does less juggling magic, and that juggling magic is actually undesirable for a heavily parallelized task.

"So there are some tasks where the Intel core will be much faster than the NVidia SM, and some tasks where the NVidia SM will be much faster."

This conversation already has a context. Arguments which ignore that context completely miss the point.

If you don't understand how I achieved the amount of processing I did, that's fine. Playing games with the semantics of a "core" somehow magically requiring all the features of current Intel-strategy chips, though, are not going to convince me.

There is more to Heaven and Earth, Horatio, than is dreamt of in Intel's philosophy. This sort of attitude towards what constitutes the no true scotsman "a real core" is why Arm is in the process of eating Intel alive, and why Tilera stands a decent chance of doing the same thing to ARM.

This is merely extreme RISC. I realize it's sort of a tradition for the modern VLIW movement to suggest that if you can't double-backflip through a flaming hoop made out of predictive NAND gates it somehow doesn't count.

But, if you actually look, the rate of modern supercomputing going to video cards is rising dramatically.

So obviously they count as cores to somebody.

You also seem to have missed the point. It's not the core scale that we're discussing here. It's the dataset scale. The number of cores you throw at a problem is not terribly important; 20 years ago it would have been breathtaking to throw 32 cores at a problem, and now that's two CPUs.

What makes an experiment cutting edge is the nature of the experiment, not the volume of hardware that you throw at it. I was talking about the /data/ and the /problem/ . Predicting movie ratings is a hell of a lot harder than feature detection.

Post reply on HN