Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MXNet – Deep Learning Framework of Choice at AWS

allthingsdistributed.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: MXNet – Deep Learning Framework of Choice at AWS

#31

Post by cs702 »

ogrisel wrote: post #26
cs702 wrote: post #23

Translation from corporatespeak: "We don't have an internally developed framework that can compete with TensorFlow, which is controlled by Google, so we are throwing our weight behind MXNet." As others have commented here, there is no evidence that MXNet is that much better (or worse) than the other frameworks.

At least MXNet is a good one that deserves more publicity and backing (in terms of maintenance effort). I find it better for the community to have AWS back a good existing open source project than to re-invent a very similar wheel one more time.

I like MXNet, and I think it's great that Amazon is backing it publicly. TensorFlow needs competition for the minds and hearts of developers.

Re: MXNet – Deep Learning Framework of Choice at AWS

#32

Post by cs702 »

werner wrote: post #28
cs702 wrote: post #23

Translation from corporatespeak: "We don't have an internally developed framework that can compete with TensorFlow, which is controlled by Google, so we are throwing our weight behind MXNet." As others have commented here, there is no evidence that MXNet is that much better (or worse) than the other frameworks.

Amazon has been building technology based on ML&DL for over 20 years and has developed several frameworks. You must have missed the announcement of this open source framework earlier in the year: https://github.com/amznlabs/amazon-dsstne .

I saw that when it was announced. DSSTNE has failed to capture the hearts and minds of developers. In my experience, it doesn't come up in any conversations about which frameworks to bet on for new product development.

And I'm rooting for Amazon (and FaceBook, and Microsoft...). TensorFlow needs competition for the hearts and minds of developers.

Re: MXNet – Deep Learning Framework of Choice at AWS

#33

Post by mli »

ogrisel wrote: post #13
oneshot908 wrote: post #2

Using 3 year-old GPUs on a much deeper network than the other guys(tm) to demonstrate awesome scaling efficiency == Intel-level FUD. Note also the absence of overall batch size. Wonder what would happen to that scaling efficiency if those GPUs were P40s? See also the absence of equivalent AlexNet numbers to further obscure attempts at comparing this to the other guys(tm). Can't wait for Intel's response to this.

What is really fishy is evaluating training time speed ups in terms of throughput. The latency induced by the parallelism mechanism (when using asynchronous data parallelism) might seriously hamper the convergence speed. The presence of this potential problem cannot be detected in the throughput metric. They should have used a convergence metric instead (e.g. training time to reach 99% of the best validation loss). I…

The results are reported using synchronized SGD with each GPU using batch-size 32. More details such as scripts to reproduce the results, scalability results on various networks (including Alexnet) and various batch sizes will be available soon. I'll put more technical details such as implementation details and performance analysis in my phd thesis.

Re: MXNet – Deep Learning Framework of Choice at AWS

#35

Post by Russell91 »

egeozcan wrote: post #34

> Machine learning (...) is being employed in a range of computing tasks where programming explicit algorithms is infeasible. I found this comment interesting. Is this really the summary of what machine learning is about?

Yes! Sometimes you know that a solution will take a particular mathematical form, without knowing what the parameters will be. So you can write down a program (function) that can express any solution of that form, and use an optimization algorithm e.g. gradient descent on labeled examples, to figure out which specific instance of your possible solutions works best.

Re: MXNet – Deep Learning Framework of Choice at AWS

#36

Post by Analog24 »

egeozcan wrote: post #34

> Machine learning (...) is being employed in a range of computing tasks where programming explicit algorithms is infeasible. I found this comment interesting. Is this really the summary of what machine learning is about?

Image classification is a classic example of such a task. How exactly would you go about writing an algorithm to tell the difference between a picture of a cat and a picture of a dog?

Re: MXNet – Deep Learning Framework of Choice at AWS

#38

Post by ogrisel »

mli wrote: post #33
ogrisel wrote: post #13

Earlier quoted context omitted.

What is really fishy is evaluating training time speed ups in terms of throughput. The latency induced by the parallelism mechanism (when using asynchronous data parallelism) might seriously hamper the convergence speed. The presence of this potential problem cannot be detected in the throughput metric. They should have used a convergence metric instead (e.g. training time to reach 99% of the best validation loss). I…

The results are reported using synchronized SGD with each GPU using batch-size 32. More details such as scripts to reproduce the results, scalability results on various networks (including Alexnet) and various batch sizes will be available soon. I'll put more technical details such as implementation details and performance analysis in my phd thesis.

Then the total batch size is growing with the number of GPUs and the convergence might be impacted both in terms of speed and solution quality (e.g. https://arxiv.org/abs/1609.04836 ).

I could believe you if tell you me that the validation loss and test accuracy of the large distributed model remains as good as the sequential, single GPU model after the same total number of epochs but this is not a given and if it's not the case I would find those benchmarks deceptive.

Re: MXNet – Deep Learning Framework of Choice at AWS

#39

Post by eva1984 »

cs702 wrote: post #23

Translation from corporatespeak: "We don't have an internally developed framework that can compete with TensorFlow, which is controlled by Google, so we are throwing our weight behind MXNet." As others have commented here, there is no evidence that MXNet is that much better (or worse) than the other frameworks.

Exactly. Among those DL frameworks, I think what TensorFlow gets right the most is the tooling support. The metric collection/visualization/checkpointing is plug-and-play in TensorFlow, others not too much. For example, summary a.k.a metric collection is just a subgraph of the whole computational graph, which can be evaluated at any time. A simple and neat abstraction indeed.

Those properties combined make TensorFlow the most engineer/practitioner friendly choice in the market. If AWS hopes to compete with TensorFlow in all seriousness, they need to catch up with support on those seeming trivial but important details.

Re: MXNet – Deep Learning Framework of Choice at AWS

#40

Post by AlexCoventry »

From the OP:

  > a Deep Learning AMI, which comes pre-installed with the popular open source
  > deep learning frameworks mentioned earlier; GPU-acceleration through CUDA
  > drivers which are already installed, pre-configured, and ready to rock
You might want to clarify that the negative reviews [0] are from earlier versions which did not include the CUDA drivers. I recently considered this AMI and rejected it for a class [1] because of these reviews.

[0] https://aws.amazon.com/marketplace/reviews/product-reviews?a...

[1] https://www.meetup.com/Cambridge-Artificial-Intelligence-Mee...

Post reply on HN