BigDL: Distributed Deep Learning on Apache Spark
1–10 of 39 posts
Re: BigDL: Distributed Deep Learning on Apache Spark
#2It does not appear that this uses a GPU at all. Which is okay of course but may not win any speed contests.
Re: BigDL: Distributed Deep Learning on Apache Spark
#3Hmm, seems interesting but wondering how it compares to H2O's Sparkling Water. Been using that for clients and i love it.
Re: BigDL: Distributed Deep Learning on Apache Spark
#4It does not appear that this uses a GPU at all. Which is okay of course but may not win any speed contests.
Intel is pushing their own chips for deep learning (i.e. Xeon and Xeon Phi). They claim that using MKL on these chips gets comparable performance to using Caffe/cuDNN on a high-end GPU.
Re: BigDL: Distributed Deep Learning on Apache Spark
#5Isn't spark more versatile than tensorflow at this point? It does graph processing and deep learning.
Plus it's built for distributed processing.
Pyspark makes it easy to use.
Re: BigDL: Distributed Deep Learning on Apache Spark
#6It does not appear that this uses a GPU at all. Which is okay of course but may not win any speed contests.
Intel is pushing their own chips for deep learning (i.e. Xeon and Xeon Phi). They claim that using MKL on these chips gets comparable performance to using Caffe/cuDNN on a high-end GPU.
Phi may yet win out, I saw a talk from an NLP researcher about Seq2Seq models and someone asked him what his wish for hardware was and it was for GPU cores to not be as stuck in lock step with each other control-wise. Not sure if he knew about the Phi's though.
Re: BigDL: Distributed Deep Learning on Apache Spark
#7It does not appear that this uses a GPU at all. Which is okay of course but may not win any speed contests.
gpu acceleration in spark in generally production ready - https://databricks.com/blog/2016/10/27/gpu-acceleration-in-d...
in fact looks like you can use tensorflow models in spark with GPU - https://databricks.com/blog/2016/12/21/deep-learning-on-data...
Re: BigDL: Distributed Deep Learning on Apache Spark
#8Isn't spark more versatile than tensorflow at this point? It does graph processing and deep learning. Plus it's built for distributed processing. Pyspark makes it easy to use.
Spark doesn't do GPU acceleration, which is super important if you don't have a lot of spare CPU capacity. The one time I tried to train a DL model on CPU it was 48x slower, and with communication overhead it would have taken more than 48 cores to match the single GPH. And given you want to do hyperparameter search on top of that, those CPU cores start adding up.
Re: BigDL: Distributed Deep Learning on Apache Spark
#9Some projects like SparkNet or DeepLearning4j+spark or even Sparkling Water are kinda doing the same thing. So, how this is compared to them?
Re: BigDL: Distributed Deep Learning on Apache Spark
#10Isn't spark more versatile than tensorflow at this point? It does graph processing and deep learning. Plus it's built for distributed processing. Pyspark makes it easy to use.
Spark doesn't do GPU acceleration, which is super important if you don't have a lot of spare CPU capacity. The one time I tried to train a DL model on CPU it was 48x slower, and with communication overhead it would have taken more than 48 cores to match the single GPH. And given you want to do hyperparameter search on top of that, those CPU cores start adding up.
that doesnt seem right. Databricks seems to have this in production. https://databricks.com/blog/2016/10/27/gpu-acceleration-in-d...
https://databricks.com/blog/2016/12/21/deep-learning-on-data...
I havent used this feature - but are you sure ?