Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BigDL: Distributed Deep Learning on Apache Spark

github.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: BigDL: Distributed Deep Learning on Apache Spark

#31

Post by vonnik »

sandGorgon wrote: post #26
agibsonccc wrote: post #20

Earlier quoted context omitted.

Chris my cofounder forgot to disclose he works on the project :). I"ll do it for him. I'd just like to say that as far as this niche is concerned. This is basically an attempt at "non gpus on spark". We are heavily biased towards cuda and distributed gpu applications: https://blogs.nvidia.com/blog/2016/10/06/how-skymind-nvidia-... I respect what intel is trying to do here, but it's going to take a lot more than "we b…

that sounds interesting. so dl4j works with spark ? https://deeplearning4j.org/spark#how is it because spark does "distributed computing" very efficiently ? In that case, would the apples-to-apples comparison be versus spark+tensorflow ? https://databricks.com/blog/2016/12/21/deep-learning-on-data...

Benchmarks we ran ourselves show that we're faster than TensorFlow using multi-GPUs for a non-trivial image processing task: https://github.com/deeplearning4j/dl4j-benchmark. That's the best apples to apples we have for the moment.

When you're aiming to put deep learning into production, a bunch of other things are important too, notably integrations. DL4J comes with integrations for Hadoop, Kafka and ElasticSearch as well as Spark. In the inference stage, we autoscale elastically as a micro-service using Lagom and a REST API. Most frameworks are just libs that don't solve problems deeper in the workflow. Our tools include data pipelines with DataVec (reusable data preprocessing), to model evaluation with Arbiter and a GUI for heuristics during training.

https://github.com/deeplearning4j/DataVec https://github.com/deeplearning4j/Arbiter https://deeplearning4j.org/visualization

Re: BigDL: Distributed Deep Learning on Apache Spark

#32

Post by deepnotderp »

agibsonccc wrote: post #20
vonnik wrote: post #11

Deeplearning4j does this already. It has a huge community, a Scala API and does model import from Keras. It's important to note that Spark is not an efficient computation later -- it's best if used for fast ETL. If you get that wrong, your going to be training slow. https://deeplearning4j.org https://github.com/deeplearning4j/ScalNet https://deeplearning4j.org/model-import-keras https://gitter.im/deeplearning4j/deepl…

Chris my cofounder forgot to disclose he works on the project :). I"ll do it for him. I'd just like to say that as far as this niche is concerned. This is basically an attempt at "non gpus on spark". We are heavily biased towards cuda and distributed gpu applications: https://blogs.nvidia.com/blog/2016/10/06/how-skymind-nvidia-... I respect what intel is trying to do here, but it's going to take a lot more than "we b…

Nothing to be "biased" about, Cuda is the industry standard.

Re: BigDL: Distributed Deep Learning on Apache Spark

#33

Post by minimaxir »

nl wrote: post #23
sandGorgon wrote: post #19

Earlier quoted context omitted.

for someone just getting started on Spark - what do you mean "somewhat adequate" ? Because I see MLLib ( https://spark.apache.org/docs/2.0.2/mllib-guide.html ) and a quick glance shows me a lot of overlap with tensorflow. At google, their graph processing system (Expander) and deep learning framework (tensorflow) are separate systems. Spark looks to be built from the graph side (RDD) first and is now getting ML compo…

So.. MLLib seems awesome, but the devil is in the detail. Example that have burnt me include things like using LogisticRegression for classification only supports binary classification, the LibSVM support only support import, the GBT implantation is weak compared to eg XGBoost etc. A lot of the time it is fine though. Graph support.. hmm. GraphX is ok, but there are lots of things that eg NetworkX has the GraphX does…

Spark 2.1.0 released this week evidentially supports multiclass logistic regression now!

Re: BigDL: Distributed Deep Learning on Apache Spark

#35

Post by nl »

sandGorgon wrote: post #29
nl wrote: post #28

Earlier quoted context omitted.

I think it's very hard to give general advice in this area. You are best off prototyping a deep spike into what you need, and seeing where things don't work. What specific graph operations do you want? If the stuff you need is there, then you might be fine! Note that the set of pre-built algorithms in GraphFrames is pretty small ( https://graphframes.github.io/user-guide.html#graph-algorith... ). It is pre-release th…

well, it is not very different from what Google Expander does - https://research.googleblog.com/2016/10/graph-powered-machin... our data sets have massively grown over the laat few months and now need a bigger solution. I think we will start off with a hosted solution like EMR - performance is not super critical right now (batch mode training)... but developer productivity is key.

Yes, certainly label propagation type algorithms are more suited to Spark than TensorFlow (although of course the fast matrix operations in TF could work well for this).

Re: BigDL: Distributed Deep Learning on Apache Spark

#36

Post by blueyes »

mmrezaie wrote: post #9

Some projects like SparkNet or DeepLearning4j+spark or even Sparkling Water are kinda doing the same thing. So, how this is compared to them?

This is a classic vanity deep-learning framework that Intel built due to NIH syndrome. It's like DSSTNE. Doomed to be abandoned. I can't a worse way to position a deep learning library than to say: this only works on CPUs. When you look at Intel's track record with software, especially their Trusted Analytics Platform this year, BigDL's prospects are poor. I'm just waiting for IBM to copy this move and come out with yet another deep learning lib: YADLL.

Re: BigDL: Distributed Deep Learning on Apache Spark

#37

Post by agibsonccc »

deepnotderp wrote: post #32
agibsonccc wrote: post #20

Earlier quoted context omitted.

Chris my cofounder forgot to disclose he works on the project :). I"ll do it for him. I'd just like to say that as far as this niche is concerned. This is basically an attempt at "non gpus on spark". We are heavily biased towards cuda and distributed gpu applications: https://blogs.nvidia.com/blog/2016/10/06/how-skymind-nvidia-... I respect what intel is trying to do here, but it's going to take a lot more than "we b…

Nothing to be "biased" about, Cuda is the industry standard.

Sure :D. It's still in my interest to disclose we partner with nvidia pretty closely though. I would hope to see competition here but we have a large vested interest in gpus succeeding. Thanks for the sentiment though!

Re: BigDL: Distributed Deep Learning on Apache Spark

#38

Post by agibsonccc »

vonnik wrote: post #31
sandGorgon wrote: post #26

Earlier quoted context omitted.

that sounds interesting. so dl4j works with spark ? https://deeplearning4j.org/spark#how is it because spark does "distributed computing" very efficiently ? In that case, would the apples-to-apples comparison be versus spark+tensorflow ? https://databricks.com/blog/2016/12/21/deep-learning-on-data...

Benchmarks we ran ourselves show that we're faster than TensorFlow using multi-GPUs for a non-trivial image processing task: https://github.com/deeplearning4j/dl4j-benchmark . That's the best apples to apples we have for the moment. When you're aiming to put deep learning into production, a bunch of other things are important too, notably integrations. DL4J comes with integrations for Hadoop, Kafka and ElasticSearch…

I have to correct chris here. He is talking about a lot of features that are in our enterprise version SKIL.

We will offer a limited developer version of SKIL for free.

Think of SKIL as similar to gitlab or github enterprise.

In SKIL we also have auto provisioning of a cluster and a higher level interface for running deep learning workloads. It auto configures most of the parameters like the spark worker native library path and setting up things like a training UI as well as installation of the mkl and cudnn libraries.

Optionally, you can also run a version of this with DC/OS and co where there is a packaged spark.

What we do have in dl4j is the raw components you can use to create these things such as datavec and dl4j-streaming which covers our integration with kafka.

Re: BigDL: Distributed Deep Learning on Apache Spark

#39

Post by deepnotderp »

agibsonccc wrote: post #37
deepnotderp wrote: post #32

Earlier quoted context omitted.

Nothing to be "biased" about, Cuda is the industry standard.

Sure :D. It's still in my interest to disclose we partner with nvidia pretty closely though. I would hope to see competition here but we have a large vested interest in gpus succeeding. Thanks for the sentiment though!

Oh I don't like the situation either, I'm just saying there's no need to be guilty! :)
Post reply on HN