MXNet – Deep Learning Framework of Choice at AWS
allthingsdistributed.com
MXNet – Deep Learning Framework of Choice at AWS
1–10 of 61 posts
Re: MXNet – Deep Learning Framework of Choice at AWS
#2Using 3 year-old GPUs on a much deeper network than the other guys(tm) to demonstrate awesome scaling efficiency == Intel-level FUD. Note also the absence of overall batch size.
Wonder what would happen to that scaling efficiency if those GPUs were P40s?
See also the absence of equivalent AlexNet numbers to further obscure attempts at comparing this to the other guys(tm).
Can't wait for Intel's response to this.
Re: MXNet – Deep Learning Framework of Choice at AWS
#3Okay, with all due respect, this is BS. I love MXNet and think it's under appreciated as well. But, pretty much its best feature is the memory mirror. (see oneshot908's comment)
Re: MXNet – Deep Learning Framework of Choice at AWS
#4Using 3 year-old GPUs on a much deeper network than the other guys(tm) to demonstrate awesome scaling efficiency == Intel-level FUD. Note also the absence of overall batch size. Wonder what would happen to that scaling efficiency if those GPUs were P40s? See also the absence of equivalent AlexNet numbers to further obscure attempts at comparing this to the other guys(tm). Can't wait for Intel's response to this.
[deleted]
Re: MXNet – Deep Learning Framework of Choice at AWS
#5Did not realize you could use MXNet declaratively (like Tensorflow/Theano) and imperatively (like Torch/Chainer). Can anyone speak more of their imperative usage of MXNet?
Re: MXNet – Deep Learning Framework of Choice at AWS
#6>MXNet can consume as little as 4 GB of memory when serving deep networks with as many as 1000 layers.
So perhaps I'm not well versed enough in deep learning, but does this mean that they solved the vanishing gradient problem? How are they managing to do this?
Re: MXNet – Deep Learning Framework of Choice at AWS
#7Did not realize you could use MXNet declaratively (like Tensorflow/Theano) and imperatively (like Torch/Chainer). Can anyone speak more of their imperative usage of MXNet?
does declaratively mean the use of expression template in C++?
I learned about it last week, I don't seem to see too much benefit if the goal is good performance.
Re: MXNet – Deep Learning Framework of Choice at AWS
#8It seems more prevalent now than it used to be, that frameworks/libraries are being used as weapons in a sort of mindshare war between the world's megacorps. Or perhaps I'm misremembering history. And I don't mean just AI; just look at Angular (Google) vs. React (Facebook).
It's a bit of a double edged sword. As developers this war gives us free access to well funded and heavily developed tools. The world has been fundamentally changed by their availability. But at the same time we need to understand that the primary reason they exist is to lock developers into a particular vendor. It's most transparent with Google's TensorFlow, where they were obvious about their intentions to offer TensorFlow services on their cloud platform.
This article more than most exemplifies their desperate attempts. For now it seems to remain mostly that, desperate attempts, with the tools remaining more-or-less platform agnostic. But I foresee a grim future where our best libraries and tools are tied inextricably to a commercial ecosystem.
Re: MXNet – Deep Learning Framework of Choice at AWS
#9>MXNet can consume as little as 4 GB of memory when serving deep networks with as many as 1000 layers . So perhaps I'm not well versed enough in deep learning, but does this mean that they solved the vanishing gradient problem? How are they managing to do this?
For deep convnets the vanishing gradient problems can mostly be solved by using residual architectures. See: https://arxiv.org/abs/1603.05027
This is kind of related to solving the vanishing gradient issue in RNNs by using additive recurrent architectures like LSTMs and GRUs.
Alternatively it's possible to use concatenative skip connections as in DenseNets: https://arxiv.org/abs/1608.06993
Still using 1000 layers is useless in practice. State of the art image classification models are in the range 30-100 layers with residual connections and varying numbers of channels per layer depending on the depth so as to keep a tractable total number of trainable parameters. The 1000 layers nets are just interesting as a memory scalability benchmark for DL frameworks and to validate empirically the feasibility of the optimization problem but are of no practical use otherwise (as far as I know).
Re: MXNet – Deep Learning Framework of Choice at AWS
#10Did not realize you could use MXNet declaratively (like Tensorflow/Theano) and imperatively (like Torch/Chainer). Can anyone speak more of their imperative usage of MXNet?
does declaratively mean the use of expression template in C++? I learned about it last week, I don't seem to see too much benefit if the goal is good performance.
No it means writing a program that defines the structure of a computation graph lazily (without executing the nodes when defining the model) so as to reuse that compute graph in a later step of the programs.
The computation graph is an in-memory datastructure that can be introspected by the program itself at runtime so as to do symbolic operations (e.g. compute the gradient of one node in the graph with respect to any ancestor input node).
theano implements this in pure Python and can generate C or CUDA code from string templates (in Python). tensorflow has to a Python API to assemble pre-built operators which are mainly written in C++ and use the Eigen linear algebra library.