Artificial Intelligence Software Is Booming, But Why Now? • Hacker News Board

Re: Artificial Intelligence Software Is Booming, But Why Now?

#61

Post by rspeer » Mon, Sep 19, 2016, 7:45 AM UTC

Xcelerate wrote: post #38

rspeer wrote: post #4

Earlier quoted context omitted.

I'm concerned about something similar, where a lot of AI techniques seem to be rushing toward a local maximum: * AI researchers do things that get good results faster out of Nvidia cards * Nvidia makes their cards faster at the things AI researchers are doing It's getting good results. We sure are going up this gradient quickly. But I don't think it's going to get us to a global maximum.

> We sure are going up this gradient quickly. But I don't think it's going to get us to a global maximum. Yes, however, this process has the nice side effect of spurring a huge interest in fundamental AI research, which will lead to alternative algorithms that ultimately perform better.

Wait, how? It seems to me that it gives people an easy way to focus on one area of AI, saying "Guess we didn't need fundamental AI research after all".

Re: Artificial Intelligence Software Is Booming, But Why Now?

#62

Post by ianai » Mon, Sep 19, 2016, 7:53 AM UTC

MichaelBurge wrote: post #51

tnecniv wrote: post #30

Earlier quoted context omitted.

There's a similar saying I like (don't know where it's from, if anywhere) "Artificial intelligence is a group of problems that we don't know how to solve. As soon as we know how to solve one, it gets a name and is no longer AI."

People say the same thing about philosophy. That once something is well-understood, it becomes a science rather than philosophy. I suppose this could be the same thing.

I always thought it was more once it got big enough.

Re: Artificial Intelligence Software Is Booming, But Why Now?

#63

Post by xyzzy123 » Mon, Sep 19, 2016, 7:57 AM UTC

gambler wrote: post #57

Houshalter wrote: post #43

Earlier quoted context omitted.

I don't buy that. Spiders can't beat a human at Go. Sure, they weren't evolved to do that. But how many generations of selective breeding do you think it would take to evolve a spider that could beat the best humans at go? Whereas if you made a spider hunting video game, I bet researchers could train AI's that could beat it within a few months. Video game playing has actually become a big area of research recently an…

Well, yes, if you could make a completely artificial, controlled environment that presents a problem well-suited for the current flavor of AI, the AI would probably be able to beat a random animal at solving that problem.

Spiders have much better hardware than the state of the art in low cost miniatuarised robotics.

However I'm not convinced that if you trained a deep net and gave it access to a spider's sensor suite and effectors, that it would do worse than an actual spider would.

A lot worse in terms of efficiency per watt, sure...

Re: Artificial Intelligence Software Is Booming, But Why Now?

#64

Post by [deleted] » Mon, Sep 19, 2016, 7:57 AM UTC

gambler wrote: post #57

Houshalter wrote: post #43

Earlier quoted context omitted.

I don't buy that. Spiders can't beat a human at Go. Sure, they weren't evolved to do that. But how many generations of selective breeding do you think it would take to evolve a spider that could beat the best humans at go? Whereas if you made a spider hunting video game, I bet researchers could train AI's that could beat it within a few months. Video game playing has actually become a big area of research recently an…

Well, yes, if you could make a completely artificial, controlled environment that presents a problem well-suited for the current flavor of AI, the AI would probably be able to beat a random animal at solving that problem.

[deleted]

Re: Artificial Intelligence Software Is Booming, But Why Now?

#65

Post by YeGoblynQueenne » Mon, Sep 19, 2016, 8:00 AM UTC

Houshalter wrote: post #43

intrasight wrote: post #40

Earlier quoted context omitted.

And then there's the metric of how pitifully little intelligence we've managed to get from all those GFLOPS. I'd say that all the GFLOPS together don't add up to the intelligence of a single Portia africana. http://news.nationalgeographic.com/2016/01/160121-jumping-sp...

I don't buy that. Spiders can't beat a human at Go. Sure, they weren't evolved to do that. But how many generations of selective breeding do you think it would take to evolve a spider that could beat the best humans at go? Whereas if you made a spider hunting video game, I bet researchers could train AI's that could beat it within a few months. Video game playing has actually become a big area of research recently an…

>> Spiders can't beat a human at Go.

Finding the best next move from a given board position at Go is a single cognitive task, whereas surviving in the natural world requires an intelligence that can excel at a possibly infinite number of such tasks.

Portia spiders are particularly amazing in that they can exhibit such complex behaviours as ambushing or stalking prey. Like all insects they do this sort of thing with a tiny number of neurons. We have no idea how to do so well with such few resources.

Re: Artificial Intelligence Software Is Booming, But Why Now?

#66

Post by dave_sullivan » Mon, Sep 19, 2016, 8:01 AM UTC

dragandj wrote: post #56

dave_sullivan wrote: post #27

Earlier quoted context omitted.

I agree w/ you about moving up a gradient quickly w/ the "GPU manufacturing deep learning research" feedback loop. I think it could last a while though. One really important area of research is figuring out how to take better advantage of greater capacity. Also, how to do more with fewer training samples (0 shot, 1 shot, etc learning). Then there's reducing the precision of the units you're using to increase capacity…

As someone who does (non-NN) ML on AMD, I might ask why you think that AMD is so far behind. In my experience, their GPU hardware is excellent, maybe even better than Nvidia, especially if you factor the price in. Where they ARE lagging heavily is: 1. GPU as a service: While all major providers (AWS, Azure) offer Teslas on their servers, there is no AMD on the cloud (that I know of) 2. Key libraries: Nvidia comes wit…

I think I agree with everything you just said.

AMD hardware is more or less the same as Nvidia; Nvidia doesn't really have any special sauces or patents that make their cards better than AMD. And you're right: in a lot of benchmarks, AMD is either faster or (at least) better-bang-for-buck than Nvidia.

Yet AMD doesn't have community support in software or cloud adoption or key libraries. The deep learning community has gone for what's easiest to work with and most readily available. Even though you might conceivably get some performance gain from AMD, it's outweighed by the amount of software that's been written in CUDA already, and how quickly that allows you to move.

Nvidia has taken an early lead and jumped on it, while AMD has only fallen further behind. I think one company just fundamentally understands the trend more than the other. But, for what is essentially hardware that answers the question, "how fast can you do matrix multiplication?", library support becomes a key differentiator. And that's where AMD is behind.

Re: Artificial Intelligence Software Is Booming, But Why Now?

#67

Post by dharma1 » Mon, Sep 19, 2016, 8:06 AM UTC

dragandj wrote: post #56

dave_sullivan wrote: post #27

Earlier quoted context omitted.

I agree w/ you about moving up a gradient quickly w/ the "GPU manufacturing deep learning research" feedback loop. I think it could last a while though. One really important area of research is figuring out how to take better advantage of greater capacity. Also, how to do more with fewer training samples (0 shot, 1 shot, etc learning). Then there's reducing the precision of the units you're using to increase capacity…

As someone who does (non-NN) ML on AMD, I might ask why you think that AMD is so far behind. In my experience, their GPU hardware is excellent, maybe even better than Nvidia, especially if you factor the price in. Where they ARE lagging heavily is: 1. GPU as a service: While all major providers (AWS, Azure) offer Teslas on their servers, there is no AMD on the cloud (that I know of) 2. Key libraries: Nvidia comes wit…

The key libraries part has kept me wondering. How much would it cost for AMD to assign a handful heavy duty engineers to this task (writing AMD optimised kernels for convolution etc)?

Their management has been fast asleep for at least 2 years

Re: Artificial Intelligence Software Is Booming, But Why Now?

#68

Post by singularity2001 » Mon, Sep 19, 2016, 8:09 AM UTC

Houshalter wrote: post #43

intrasight wrote: post #40

Earlier quoted context omitted.

And then there's the metric of how pitifully little intelligence we've managed to get from all those GFLOPS. I'd say that all the GFLOPS together don't add up to the intelligence of a single Portia africana. http://news.nationalgeographic.com/2016/01/160121-jumping-sp...

I don't buy that. Spiders can't beat a human at Go. Sure, they weren't evolved to do that. But how many generations of selective breeding do you think it would take to evolve a spider that could beat the best humans at go? Whereas if you made a spider hunting video game, I bet researchers could train AI's that could beat it within a few months. Video game playing has actually become a big area of research recently an…

Build us a machine which can navigate the environment as swiftly as a spider, THEN general spider AI is reached. ( not talking about the hardware, talking about the smooth integration of image recognition, motor control, general learning abilities and whatnot)

Re: Artificial Intelligence Software Is Booming, But Why Now?

#69

Post by Houshalter » Mon, Sep 19, 2016, 8:10 AM UTC

PieterH wrote: post #50

Houshalter wrote: post #37

Date Approximate cost per GFLOPS inflation adjusted to 2013 US dollars --------------------------------------------------------------------------------- 1961 $8.3 trillion 1984 $42,780,000 1997 $42,000 April 2000 $1,300 May 2000 $836 August 2003 $100 August 2007 $52 March 2011 $1.80 August 2012 $0.73 June 2013 $0.22 November 2013 $0.16 December 2013 $0.12 January 2015 $0.06

Nice chart. I call this exponential drop "cost gravity". 50% every 18 months, more or less.

It's about 13 months per halving. It seems to be consistent whether I start at 2000 or 1961.

Re: Artificial Intelligence Software Is Booming, But Why Now?

#70

Post by dharma1 » Mon, Sep 19, 2016, 8:11 AM UTC

jhj wrote: post #13

hooloovoo_zoo wrote: post #7

Earlier quoted context omitted.

Convolutional layers have been around longer than that and don't have much to do with the vanishing gradient problem, which was never that big a problem to begin with. ReLUs are helpful, but they do not lead to THAT big of a speedup. Convolutional layers have definitely had a big impact for accuracy in image problems and sound problems. I think #2 is very important; they speed up both training and evaluation by a fac…

> they speed up both training and evaluation by a factor of 100 The factor is more like 5-10x for truly optimized CPU versus GPU in fp32 (e.g., nnpack on CPU versus cuDNN).

nnpack is excellent - but also very new. Noticed it just got NEON optimisations for Arm- do you know if it's possible to use it together with OpenCL on mobile for combined CPU/GPU inference?