Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building high-level features using large scale unsupervised learning

research.google.com

Post reply on HN
Print view View original

111–120 of 196 posts

Re: Building high-level features using large scale unsupervised learning

#111

Post by ippisl »

jules wrote: post #94
seiji wrote: post #76

16,000 cores sounds impressive until you realize it's just five to ten modern GPUs. For Google, it's easier to just run a 1,000 machine job than requisition some GPUs. See: http://www.nvidia.com/object/tesla-servers.html (4.5 teraflops in one card ) Reminder: GPUs will destroy the world.

The problem is not the theoretical peak teraflops. The problem is actually achieving those teraflops with useful work. Due to architecture that is easier on a CPU than on a GPU, so you can't directly compare teraflops and conclude that GPUs are superior. Getting something to run fast on a GPU is very difficult. And actually the thing that does 4.5 teraflops in single precision does only 95 gigaflops in double precisi…

Single precision is probably fine for a neural network. Neural networks are somewhat insensitive to noise and failure and single precision adds very little noise.

Re: Building high-level features using large scale unsupervised learning

#112

Post by coldarchon »

nkoren wrote: post #104
JohnHaugeland wrote: post #91

Earlier quoted context omitted.

"It does this for 20,000 different objects categories - this is getting close to matching human visual ability" No, it isn't. This classifier cannot identify theme variations, unknown rotations, will confuse new objects for objects it already knows, is unable to cope with camera distortion, needs fixed lighting, has no capacity for weather, does not work in the time you need to run away from a tiger, requires hundred…

I'm not a Machine Learning / AI expert, so I have to ask: if running a neural network on 16,000 cores with a training set of 10 million objects isn't cutting edge research -- and if running "far larger" networks than this, as you say you have, also isn't cutting edge research -- then please tell me: what is cutting edge research? I ask this question in all seriousness; I'd really like to know. (And yes, I see that yo…

There are projects which connect computers during the downtimes of corporations. They don't need all their power at night and offer this to open projects of distributed neural nets. The smallest one has 25.000 cores ..

Re: Building high-level features using large scale unsupervised learning

#113

Post by JohnHaugeland »

nkoren wrote: post #104
JohnHaugeland wrote: post #91

Earlier quoted context omitted.

"It does this for 20,000 different objects categories - this is getting close to matching human visual ability" No, it isn't. This classifier cannot identify theme variations, unknown rotations, will confuse new objects for objects it already knows, is unable to cope with camera distortion, needs fixed lighting, has no capacity for weather, does not work in the time you need to run away from a tiger, requires hundred…

I'm not a Machine Learning / AI expert, so I have to ask: if running a neural network on 16,000 cores with a training set of 10 million objects isn't cutting edge research -- and if running "far larger" networks than this, as you say you have, also isn't cutting edge research -- then please tell me: what is cutting edge research? I ask this question in all seriousness; I'd really like to know. (And yes, I see that yo…

I'm his son.

Let's take the example of The Netflix Prize, a $1 million bounty that the movie shipping organization ran several years ago. Their purpose was to improve their ratings prediction algorithm, under the pretext that people frequently ran out of ideas of what to rent, and that a successful suggestion algorithm would keep people as customers longer after that point.

So, they carefully defined the success rate of their algorithm - that is, make it predict some set of actually-rated movies X on a 1-5 half-integer scale, take the arithmetic mean of (the sum of (the square of each error from the real rating)) - which we'll call root mean square error, or RMSE - and you have your "score," where towards zero is perfect.

Their predictor had a score of I think 0.973 something (it's been years, don't quote me on that.) Their challenge was simple.

Beat their score by ten percent, and you trigger a one month end-of-game. At the end of that month, whoever's best wins le prize. One million dollars, obligatory Doctor Evil finger and all.

Netflix provided (anonymized) a little over 100 million actual ratings, where all you had was a userID, a movieID, a real rating, and separately, a mapping "this movieID is this title." You were only allowed to use datasets in your solution that were freely available to everybody, and you had to reveal them and write a paper about your strategy within one month after you accept le prize, honh honh honh.

Seriously, it was awesome. They were going to do a second one, but lawyers, and the world sadded.

So, there, you've got a ten times larger dataset. So surely sixteen thousand cores is the drastic thing, right?

Well, not really. I was running my solution on 32 Teslas, which in the day were $340 in bulk and had 480 cores each. So I actually "only" had 15,360 cores, which falls a whopping four percent short of Google's approach, which several years ago cost me about the price of a recently used car, and which I was able to resell afterwards as used, but without the bulk discount, for almost exactly what I paid for them in the first place.

Swish.

And I mean, I've got to imagine that someone else chasing that million dollar prize who thought they were going to get it invested more than I did. There were groups of up to a dozen people, data mining companies, etc.

So if one dude sitting in his then-Boise apartment can spend like $11k on a ten times this dataset dataset over a commercial prize?

Yeah.

Cyc still pantses all of us.

Re: Building high-level features using large scale unsupervised learning

#114

Post by scotty79 »

Dn_Ab wrote: post #101
scotty79 wrote: post #66

Earlier quoted context omitted.

I strongly doubt that. I think babies have much richer input than any CV system up to date. In my opinion movement is crucial for understanding image.

A human baby learns in a very interesting way. By using an overcomplete basis to sparsely code data. This method has only recently started getting attention in ML. A human baby learns from uncleaned raw data using far less energy with better generalization than a computer and fuses large amount of data without suffering from dimensionality curses. I think it is safe to say that human babies are still ahead. for now.

I agree that CV is currently behind human sight. I just argue that this is more due to lacking input processing than to actual classification engine. The reason they used 200x200 px images, I think, was that bigger images couldn't be analysed in sufficient quantity.

In a way their first input processing of the real images was pixelisation filter. If you feed pixelised image to a person you see how much information is lost. If you make single pixels occupy significant portion of view person might loose ability to recognize the image at all. Feeding so little information to CV system is like trying to teach nearly blind man to see.

To improve CV we should focus on finding best ways of converting full resolution visual data to something of smaller volume in such way that important features are preserved. This input data IMO should also include time. I, thanks to crappy eyesight often recognize people, actions, objects relying more on how they move not how they look to me. Even with sharp eysight sometime your vision just gets stuck and can't recognize what is in the scene you are currently looking at. You can't understand what you see until something in the scene moves or you move a bit.

Re: Building high-level features using large scale unsupervised learning

#115

Post by Dn_Ab »

JohnHaugeland wrote: post #93
jpeterson wrote: post #5

Maybe I'm missing something here, but how exactly is it "unlabeled" data if they're specifically feeding it millions of pictures of "faces"? I mean, if you make a specific selection of the type of images you train the network on, isn't that basically equivalent to labeling them?

You're correct: it isn't an unlabelled system, and the article author is deeply confused about basic topics in artificial intelligence. What he's trying to talk about is "this is an unsupervised feature detector in a large dataset which is only categorized, and where no human has provided correct answers up front to verify progress." The reason this matters (and it doesn't matter very much) is that that means that in…

The article authors of the paper? How can you say they are deeply confused - have you not seen their previous work and presentations? Everything else you say I agree with.

Re: Building high-level features using large scale unsupervised learning

#116

Post by Symmetry »

JohnHaugeland wrote: post #87

The singularity is a poorly constructed myth. It is built around the presumption that intelligence is a linear function of CPU power, and that surely as CPU power rises, so shall intelligence; the problem is, that prediction was made in the 1970s, since which CPU power has risen ten decimal orders of magnitude, and we still don't have much better speech recognition than we did back then, let alone anything even appro…

Well, maybe. There are a whole lot of very different things called "The Singularity" and some of them are much more reasonable than others.

There's the Cambpellian Singularity, which says that we won't be able to predict what will happen next. Pretty non-controversial as far as it goes.

There's the Vingean Singularity, which says that if we ever develop AIs that can think as fast and as well as humans then due to Moore's Law they'll be thinking twice as fast as humans after 2 years, so they'll start designing chips and the period of Moore's law will fall to 1 year, and so on with us reaching infinite computing power in finite time. I think this vision is flawed.

Relatedly, there's the Intelligence Explosion Singularity (associated with Yudkowsky), which says that as soon as its AIs designing AIs, smarter AIs will relativly quickly be able to make even smarter AIs and we'll get a "fwoosh" effect, though not to infinity in finite time. I find this unlikely, but can't rule it out.

There's one I don't have a handy name for, but lets call it the AI Revolution viewpoint, which is that AIs will cause civilization to switch to a faster mode of progress, just like the Agricultural Revolution and Industrial Revolution did. This one will only look like a singularity in hindsight, and might seem gradual to the people living through it. I think this one is pretty credible.

There's the Kurzweilian Singularity, where thanks to Accelerating Change we'll someday pass a point which will arbitrarily be called the Singularity. As far as I can tell this is Kurzweil appropriating the hot word of the moment for his ideas a la Javascript.

Then there's the Naive Singularity, which equates processing power with intelligence and then concludes that computers must be getting smarter. This is indeed totally naive and not something we should worry about. I guess the linked paper is evidence that you can substitute a faster computer for smarter AI researchers to some extent, but probably not a very large one.

Re: Building high-level features using large scale unsupervised learning

#117

Post by JohnHaugeland »

eric_bullington wrote: post #81
Homunculiheaded wrote: post #53

There was an interesting discussion on Quora about this recently[0] The most relevant quote being perhaps: "The magic of the brain is not the number of neurons, but how the circuits are wired and how they function dynamically. If you put 1 billion transistors together, you don't get a functioning CPU. And if you put 100 billion neurons together, you don't get an intelligent brain." 0. http://www.quora.com/How-big-is-…

That's an interesting discussion, but this experiment suggests exactly the opposite (perhaps that's why you included the discussion). Who knows, if we put 1 billion cores together, and fed it a massive amount of data (akin to what a baby receives as he/she matures), perhaps we would get a brain we would consider "intelligent". The fact that this system was able to pick out high-level features like "face" and "cat" wi…

"That's an interesting discussion, but this experiment suggests exactly the opposite (perhaps that's why you included the discussion)."

It absolutely does not. This experiment supports that position strongly.

What this experiment shows is that said meaningful structure can be progressively, automatically discovered.

"and with only 1000 cores, not 1 billion"

Comparing CPU cores to individual neurons is more than slightly disingenuous.

Re: Building high-level features using large scale unsupervised learning

#118

Post by JohnHaugeland »

Symmetry wrote: post #109
seiji wrote: post #76

16,000 cores sounds impressive until you realize it's just five to ten modern GPUs. For Google, it's easier to just run a 1,000 machine job than requisition some GPUs. See: http://www.nvidia.com/object/tesla-servers.html (4.5 teraflops in one card ) Reminder: GPUs will destroy the world.

What a GPU calls a "core" doesn't at all correspond to what a CPU calls a "core". Going by the CPU definition (something like "something that can schedule memory accesses") a high end GPU will only have 60 or so cores. And going by the the GPU definition (An execution unit) a high end CPU will tend to have 30-something cores. GPUs do have fundamentally more execution resources, but that comes at a price and not every…

"What a GPU calls a "core" doesn't at all correspond to what a CPU calls a "core"."

They aren't as different as you imagine. They're general purpose programmable arithmetic units with processing rates on the order of 20-30% of CPUs, provided the limitation that they're all doing roughly the same thing.

For most machine learning tasks, that's exactly what you're doing anyway. Oh no, your neural network engine has to be parallel? C'est damage!

Re: Building high-level features using large scale unsupervised learning

#119

Post by coldarchon »

marshallp wrote: post #32
forgotusername wrote: post #6

Can we perhaps edit "singularity is near" out of the title? This sounds impressive, but having a bunch of racks able to classify the outline of a face is vastly disconnected from machine and humanity merging.

I put the singularity bit in to make it relevant for those who are non-technical. This experiment is significant because it shows that large artificial neural networks can be made to work. People have tried and failed at this for decades. This technigue was "discovered" by geoff hinton at the university of toronto in 2005. However, nobody at tried (or maybe got enough funds) to try it this scale. If this continues to…

You have absolutely no idea I guess.

Let's have a look at the human brain. 1.000.000.000 neuron cells in the brain which can have 200.000 connections with 2 different states. The number of different states of the brain exceeds the number of atoms in the universe, we are speaking of 2 to the power of 200.000.000.000.000 compared to 10 to the power of 82. I guess you can calculate this, right?

And you compare this to 16.000 cores?

Re: Building high-level features using large scale unsupervised learning

#120

Post by JohnHaugeland »

saalweachter wrote: post #106
JohnHaugeland wrote: post #89

Earlier quoted context omitted.

Unsupervised training is not particularly significant, and was the original form of neural network. "our experimental results reveal" No, they don't. We had one of these in the 1980s.

Did we? A lot of early works in AI were ... /overstated/.[1] While a lot of concepts were created way back when, a lot of results weren't really generated. It's extremely valuable for someone to actually go and do a thing, now that we can, even if someone had the idea for the thing eons ago. [1] http://dl.acm.org/citation.cfm?id=1045340

The early works in AI with regards to unsupervised learning were in the 1940s and 1950s. Claude Shannon had demonstrated a chess learning system which taught itself by playing him to defeat him in under two weeks as early as 1949.

No, they weren't overstated. They were hyped by a clueless press. There's a pretty critical difference. It's a bit like how the early web pioneers didn't say that the web was going to revolutionize the delivery of dog food; it was a journalist who said that.

"It's extremely valuable for someone to actually go and do a thing, now that we can"

Self organized unsupervised learning was in use for optical classification of potatoes in the feeding of Frito Lay automated processing plants in the late 1970s.

Please distinguish that you haven't actually looked for earlier examples from that you imagine none exist. Thanks.

Post reply on HN