[offtopic] I think presentations with ascending bar charts are sort of cliche.
MXNet – Deep Learning Framework of Choice at AWS
41–50 of 61 posts
Re: MXNet – Deep Learning Framework of Choice at AWS
#42Did not realize you could use MXNet declaratively (like Tensorflow/Theano) and imperatively (like Torch/Chainer). Can anyone speak more of their imperative usage of MXNet?
it means you can declare gpu array like those in numpy/torch, write them imperatively from python side, and mix them with the graph computation, instead of forcing everything to be part of a graph
Re: MXNet – Deep Learning Framework of Choice at AWS
#43Translation from corporatespeak: "We don't have an internally developed framework that can compete with TensorFlow, which is controlled by Google, so we are throwing our weight behind MXNet." As others have commented here, there is no evidence that MXNet is that much better (or worse) than the other frameworks.
There is a huge distributed performance advantages vs TensorFlow. You can get a hint from Prof. Carlos Guestrin's keynote talk at Data Science Summit 2016. Also, CMU CS Dean Andrew Moore cited MXNet as "is the most scalable framework for deep learning I have seen"
Re: MXNet – Deep Learning Framework of Choice at AWS
#44Earlier quoted context omitted.
What is really fishy is evaluating training time speed ups in terms of throughput. The latency induced by the parallelism mechanism (when using asynchronous data parallelism) might seriously hamper the convergence speed. The presence of this potential problem cannot be detected in the throughput metric. They should have used a convergence metric instead (e.g. training time to reach 99% of the best validation loss). I…
The results are reported using synchronized SGD with each GPU using batch-size 32. More details such as scripts to reproduce the results, scalability results on various networks (including Alexnet) and various batch sizes will be available soon. I'll put more technical details such as implementation details and performance analysis in my phd thesis.
Are claiming MXNet gets a ~109x speedup in training time to X% accuracy with synchronized SGD on 128 gpus?
Re: MXNet – Deep Learning Framework of Choice at AWS
#45Earlier quoted context omitted.
Vanishing gradient isn't the same as memory efficiency. The memory mirror option is what allows this extremely efficient memory usage by only being 30% more compute intensive.
Yes, but that's not what I asked about.
Vanishing gradient is solved using model architectural choices: ReLu activation instead of sigmoid or tanh, using batch-normalization, using LSTMs
These are orthogonal to memory management and neural net framework choices.
Re: MXNet – Deep Learning Framework of Choice at AWS
#46It seems more prevalent now than it used to be, that frameworks/libraries are being used as weapons in a sort of mindshare war between the world's megacorps. Or perhaps I'm misremembering history. And I don't mean just AI; just look at Angular (Google) vs. React (Facebook). It's a bit of a double edged sword. As developers this war gives us free access to well funded and heavily developed tools. The world has been fu…
Then utilize torch.
Isn't Torch actively supported by Facebook? https://research.facebook.com/research/torch/
Re: MXNet – Deep Learning Framework of Choice at AWS
#47Translation from corporatespeak: "We don't have an internally developed framework that can compete with TensorFlow, which is controlled by Google, so we are throwing our weight behind MXNet." As others have commented here, there is no evidence that MXNet is that much better (or worse) than the other frameworks.
There is a huge distributed performance advantages vs TensorFlow. You can get a hint from Prof. Carlos Guestrin's keynote talk at Data Science Summit 2016. Also, CMU CS Dean Andrew Moore cited MXNet as "is the most scalable framework for deep learning I have seen"
This recent OSDI paper [1] has a direct comparison in Fig 8. It appears there is no particularly pronounced distribution or general performance advantage, and TensorFlow actually outperforms MXNet in this comparison.
1: https://www.usenix.org/system/files/conference/osdi16/osdi16...
[full disclosure, I work on the TensorFlow team]
Re: MXNet – Deep Learning Framework of Choice at AWS
#48Earlier quoted context omitted.
The results are reported using synchronized SGD with each GPU using batch-size 32. More details such as scripts to reproduce the results, scalability results on various networks (including Alexnet) and various batch sizes will be available soon. I'll put more technical details such as implementation details and performance analysis in my phd thesis.
Then the total batch size is growing with the number of GPUs and the convergence might be impacted both in terms of speed and solution quality (e.g. https://arxiv.org/abs/1609.04836 ). I could believe you if tell you me that the validation loss and test accuracy of the large distributed model remains as good as the sequential, single GPU model after the same total number of epochs but this is not a given and if it's…
There are a lot of papers talking about the trade-off between algorithm convergence (validation accuracy) and system efficiency. At least it is my major phd research topic at CMU. In the context of synchronized SGD on deep CNN models, my observation is that up to batch-size X, the convergence speed is not so sensitive to the batch size; between X and Y, we still get good convergence rate by tuning the hyper-paramters carefully; but beyond Y, it then becomes an interesting research question.
Both X and Y are related to the dataset and network complexity. A rough guess I often use is num_classes num_classes and Y ~= 10X. To accelerate the convergence for batch size between X and Y, we can either increase the data augmentation or learning rate, or both. The basic idea is to add more noise to the SGD training to avoid falling into suboptimal points too easily.
The paper you mentioned studies the extremely case that batch size >> Y. They used CIFAR 10 (num_classes = 10) and batch size (20% num_examples = 12K). I also surprised that they also extended our earlier work to CNN and showed promising results (Sec 4.2)
But also as mentioned by the paper authors, there is little theory we can say about that. I expected that the research community will have fun about it for a while.
But back to the MXNet benchmark, we did successfully tuned the hyper-parameters with 128 GPUs and batch size = 32 * 128 to match the convergence compared to a single machine on the Imagenet 1K dataset. So we think our setting is reasonable. But the main point here is that we are more willing to show how fast the system can achieve, so that researchers can easier try more efficient distributed algorithms here.
Re: MXNet – Deep Learning Framework of Choice at AWS
#49Li Mu, the core developer behind MXNet, works for Amazon recently.
Re: MXNet – Deep Learning Framework of Choice at AWS
#50Earlier quoted context omitted.
There is a huge distributed performance advantages vs TensorFlow. You can get a hint from Prof. Carlos Guestrin's keynote talk at Data Science Summit 2016. Also, CMU CS Dean Andrew Moore cited MXNet as "is the most scalable framework for deep learning I have seen"
This recent OSDI paper [1] has a direct comparison in Fig 8. It appears there is no particularly pronounced distribution or general performance advantage, and TensorFlow actually outperforms MXNet in this comparison. 1: https://www.usenix.org/system/files/conference/osdi16/osdi16... [full disclosure, I work on the TensorFlow team]
The version tested in paper should not have P2P,so it was much slower than current version.