Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MXNet – Deep Learning Framework of Choice at AWS

allthingsdistributed.com

Post reply on HN
Print view View original

11–20 of 61 posts

Re: MXNet – Deep Learning Framework of Choice at AWS

#11

Post by piiswrong »

oneshot908 wrote: post #2

Using 3 year-old GPUs on a much deeper network than the other guys(tm) to demonstrate awesome scaling efficiency == Intel-level FUD. Note also the absence of overall batch size. Wonder what would happen to that scaling efficiency if those GPUs were P40s? See also the absence of equivalent AlexNet numbers to further obscure attempts at comparing this to the other guys(tm). Can't wait for Intel's response to this.

Amazon probably used P2 because they want to advertise it. We can get almost linear speedup on 10 8xM40 machines using MXNet. Batch size is linearly increased with # of machines but empirically it doesn't hurt convergence, at least on imagenet.

I mean who cares about AlexNet any more? It's 2016 already. It trains in under 2h on a single machine. Distributing it doesn't make much sense

Re: MXNet – Deep Learning Framework of Choice at AWS

#12

Post by billconan »

ogrisel wrote: post #10
billconan wrote: post #7

Earlier quoted context omitted.

does declaratively mean the use of expression template in C++? I learned about it last week, I don't seem to see too much benefit if the goal is good performance.

No it means writing a program that defines the structure of a computation graph lazily (without executing the nodes when defining the model) so as to reuse that compute graph in a later step of the programs. The computation graph is an in-memory datastructure that can be introspected by the program itself at runtime so as to do symbolic operations (e.g. compute the gradient of one node in the graph with respect to an…

"defines structure of a computation graph lazily (without executing the nodes when defining the model)"

But this sounds exactly like expression template.

Re: MXNet – Deep Learning Framework of Choice at AWS

#13

Post by ogrisel »

oneshot908 wrote: post #2

Using 3 year-old GPUs on a much deeper network than the other guys(tm) to demonstrate awesome scaling efficiency == Intel-level FUD. Note also the absence of overall batch size. Wonder what would happen to that scaling efficiency if those GPUs were P40s? See also the absence of equivalent AlexNet numbers to further obscure attempts at comparing this to the other guys(tm). Can't wait for Intel's response to this.

What is really fishy is evaluating training time speed ups in terms of throughput. The latency induced by the parallelism mechanism (when using asynchronous data parallelism) might seriously hamper the convergence speed. The presence of this potential problem cannot be detected in the throughput metric. They should have used a convergence metric instead (e.g. training time to reach 99% of the best validation loss).

If they can achieve 109x speed up with 128 GPUs using synchronous data parallelism with a batch size tuned for optimal single GPU convergence time, then this is very impressive (but quite unlikely).

However I don't think that publishing training benchmarks on Inception v3 (vs say AlexNet) is a fraud. Inception v3 is close to the state of the art and very good at using few parameters & inference FLOPS for a good test accuracy.

Inception v3 has been publicly available for quite a long time in a variety of DL toolkits along with pre-trained weights.

Re: MXNet – Deep Learning Framework of Choice at AWS

#14

Post by p1esk »

piiswrong wrote: post #11
oneshot908 wrote: post #2

Using 3 year-old GPUs on a much deeper network than the other guys(tm) to demonstrate awesome scaling efficiency == Intel-level FUD. Note also the absence of overall batch size. Wonder what would happen to that scaling efficiency if those GPUs were P40s? See also the absence of equivalent AlexNet numbers to further obscure attempts at comparing this to the other guys(tm). Can't wait for Intel's response to this.

Amazon probably used P2 because they want to advertise it. We can get almost linear speedup on 10 8xM40 machines using MXNet. Batch size is linearly increased with # of machines but empirically it doesn't hurt convergence, at least on imagenet. I mean who cares about AlexNet any more? It's 2016 already. It trains in under 2h on a single machine. Distributing it doesn't make much sense

2 hours to train Alexnet on a single machine? Link please.

Re: MXNet – Deep Learning Framework of Choice at AWS

#15

Post by ogrisel »

billconan wrote: post #12
ogrisel wrote: post #10

Earlier quoted context omitted.

No it means writing a program that defines the structure of a computation graph lazily (without executing the nodes when defining the model) so as to reuse that compute graph in a later step of the programs. The computation graph is an in-memory datastructure that can be introspected by the program itself at runtime so as to do symbolic operations (e.g. compute the gradient of one node in the graph with respect to an…

"defines structure of a computation graph lazily (without executing the nodes when defining the model)" But this sounds exactly like expression template.

But neither declarative DL toolkits (theano & tensorflow) use that C++ language feature: the computation graph is typically defined by writing a python script to assembles building blocks dynamically at runtime.

Once the graph is defined, it can be passed along with concrete values for the input nodes to the runtime framework to execute the section of the graph of interest (possibly with code generation + compilation).

Re: MXNet – Deep Learning Framework of Choice at AWS

#16

Post by deepnotderp »

fpgaminer wrote: post #8

It seems more prevalent now than it used to be, that frameworks/libraries are being used as weapons in a sort of mindshare war between the world's megacorps. Or perhaps I'm misremembering history. And I don't mean just AI; just look at Angular (Google) vs. React (Facebook). It's a bit of a double edged sword. As developers this war gives us free access to well funded and heavily developed tools. The world has been fu…

Then utilize torch.

Re: MXNet – Deep Learning Framework of Choice at AWS

#17

Post by deepnotderp »

bsfjgngdnxy wrote: post #6

>MXNet can consume as little as 4 GB of memory when serving deep networks with as many as 1000 layers . So perhaps I'm not well versed enough in deep learning, but does this mean that they solved the vanishing gradient problem? How are they managing to do this?

Vanishing gradient isn't the same as memory efficiency. The memory mirror option is what allows this extremely efficient memory usage by only being 30% more compute intensive.

Re: MXNet – Deep Learning Framework of Choice at AWS

#18

Post by imh »

This reads weirdly. He talks about how MXNet is the best choice without comparing it to other frameworks. That's the whole point of choosing between things. I'm sure they did the legwork to make this decision, and some insight into that choice might help others follow. Without that, my distrust radar is blinking.

Re: MXNet – Deep Learning Framework of Choice at AWS

#20

Post by piiswrong »

p1esk wrote: post #14
piiswrong wrote: post #11

Earlier quoted context omitted.

Amazon probably used P2 because they want to advertise it. We can get almost linear speedup on 10 8xM40 machines using MXNet. Batch size is linearly increased with # of machines but empirically it doesn't hurt convergence, at least on imagenet. I mean who cares about AlexNet any more? It's 2016 already. It trains in under 2h on a single machine. Distributing it doesn't make much sense

2 hours to train Alexnet on a single machine? Link please.

https://developer.nvidia.com/cudnn Alex did it on 2x580 in 2012. Took him 1 week. It's 60x faster now even compared to K40
Post reply on HN