21–30 of 61 posts
>MXNet can consume as little as 4 GB of memory when serving deep networks with as many as 1000 layers . So perhaps I'm not well versed enough in deep learning, but does this mean that they solved the vanishing gradient problem? How are they managing to do this?
Vanishing gradient isn't the same as memory efficiency. The memory mirror option is what allows this extremely efficient memory usage by only being 30% more compute intensive.
>MXNet can consume as little as 4 GB of memory when serving deep networks with as many as 1000 layers . So perhaps I'm not well versed enough in deep learning, but does this mean that they solved the vanishing gradient problem? How are they managing to do this?
For deep convnets the vanishing gradient problems can mostly be solved by using residual architectures. See: https://arxiv.org/abs/1603.05027 This is kind of related to solving the vanishing gradient issue in RNNs by using additive recurrent architectures like LSTMs and GRUs. Alternatively it's possible to use concatenative skip connections as in DenseNets: https://arxiv.org/abs/1608.06993 Still using 1000 layers is…
As others have commented here, there is no evidence that MXNet is that much better (or worse) than the other frameworks.
Using 3 year-old GPUs on a much deeper network than the other guys(tm) to demonstrate awesome scaling efficiency == Intel-level FUD. Note also the absence of overall batch size. Wonder what would happen to that scaling efficiency if those GPUs were P40s? See also the absence of equivalent AlexNet numbers to further obscure attempts at comparing this to the other guys(tm). Can't wait for Intel's response to this.
MXNet is the only deep learning framework that has proper support for R. That's why I use it and it is pretty nice IMO.
Translation from corporatespeak: "We don't have an internally developed framework that can compete with TensorFlow, which is controlled by Google, so we are throwing our weight behind MXNet." As others have commented here, there is no evidence that MXNet is that much better (or worse) than the other frameworks.
Using 3 year-old GPUs on a much deeper network than the other guys(tm) to demonstrate awesome scaling efficiency == Intel-level FUD. Note also the absence of overall batch size. Wonder what would happen to that scaling efficiency if those GPUs were P40s? See also the absence of equivalent AlexNet numbers to further obscure attempts at comparing this to the other guys(tm). Can't wait for Intel's response to this.
Comparisons on AlexNet are not very useful now. I can get AlexNet-like quality a lot cheaper (at test time) now, and for the same computational cost I can get a lot better in quality of results ... or even better if I accept more cost. I can't think of a good reason to evaluate AlexNet nowadays, I'm more annoyed at the other guys(tm) that (exclusively) do, since that means to get meaningful datapoints I need to rerun…
A platform that runs AlexNet well has excellent computation performance for the convolution layers but it also has excellent algorithms/communication for parallelizing the model/data by whatever means.
Networks that attempt to minimize computation and/or communication are cool, but they should be considered in that light IMO.
It's also a great estimate of the low-end for strong scaling. There's a lot of bread and butter machine learning at this level in my experience.
Translation from corporatespeak: "We don't have an internally developed framework that can compete with TensorFlow, which is controlled by Google, so we are throwing our weight behind MXNet." As others have commented here, there is no evidence that MXNet is that much better (or worse) than the other frameworks.
Using 3 year-old GPUs on a much deeper network than the other guys(tm) to demonstrate awesome scaling efficiency == Intel-level FUD. Note also the absence of overall batch size. Wonder what would happen to that scaling efficiency if those GPUs were P40s? See also the absence of equivalent AlexNet numbers to further obscure attempts at comparing this to the other guys(tm). Can't wait for Intel's response to this.
Amazon probably used P2 because they want to advertise it. We can get almost linear speedup on 10 8xM40 machines using MXNet. Batch size is linearly increased with # of machines but empirically it doesn't hurt convergence, at least on imagenet. I mean who cares about AlexNet any more? It's 2016 already. It trains in under 2h on a single machine. Distributing it doesn't make much sense
Amazon is at its best when it's customer obsessed and at its worst when it puts politics first.
All IMO of course.