Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BigDL: Distributed Deep Learning on Apache Spark

github.com

Post reply on HN
Print view View original

21–30 of 39 posts

Re: BigDL: Distributed Deep Learning on Apache Spark

#21

Post by agibsonccc »

sandGorgon wrote: post #5

Isn't spark more versatile than tensorflow at this point? It does graph processing and deep learning. Plus it's built for distributed processing. Pyspark makes it easy to use.

Spark is vastly different. Tensorflow focuses more on numerical computing and is a low level tool.

Spark is more focused on "counting at scale with a functional DSL". Hence its focus on things like ETL and columnar processing ala dataframes.

As far as spark doing "deep learning" what you should mean here is: "libraries in the ecosystem leverage spark as a data access layer for doing the real numerical compute"

Spark can count things with functional programming. It's not meant for heavy numerical operations. They are working on this where they can but you really can't beat a gpu or good ole simd instructions on hardware.

Re: BigDL: Distributed Deep Learning on Apache Spark

#22

Post by agibsonccc »

sandGorgon wrote: post #7
rustyconover wrote: post #2

It does not appear that this uses a GPU at all. Which is okay of course but may not win any speed contests.

gpu acceleration in spark in generally production ready - https://databricks.com/blog/2016/10/27/gpu-acceleration-in-d... in fact looks like you can use tensorflow models in spark with GPU - https://databricks.com/blog/2016/12/21/deep-learning-on-data...

Tensorframes despite the marketing is already defunct. It hasn't seen a commit since august.

https://github.com/databricks/tensorframes

Spark is just a data access layer here. It's not even remotely gpu friendly. Most people also still relies on mesos or yarn for running distributed. The library you're using matters alot. Mesos just added gpu support: http://mesos.apache.org/documentation/latest/gpu-support/

Yarn can sorta support it with node labeling for job completion but it's still kinda hacky.

The real work in this space (without the marketing) is done by IBM: http://www.slideshare.net/ishizaki/exploiting-gpus-in-spark

When spark can (without "production ready" buzzwords) run gpus like this out of the box then we're talking. For now spark needs a companion library to work with gpus though.

Re: BigDL: Distributed Deep Learning on Apache Spark

#23

Post by nl »

sandGorgon wrote: post #19
nl wrote: post #12

Earlier quoted context omitted.

(Heavy Spark user here) Comparing Spark and TensorFlow is sort of like comparing Numpy and Pandas. There is some overlap, but they are pretty different things. Spark is a big data manipulation tool, which comes with a somewhat-adequate machine learning library. TensorFlow is an optimised math library with machine learning operations built on it. Spark doesn't support GPU operations (although as you note Databricks ha…

for someone just getting started on Spark - what do you mean "somewhat adequate" ? Because I see MLLib ( https://spark.apache.org/docs/2.0.2/mllib-guide.html ) and a quick glance shows me a lot of overlap with tensorflow. At google, their graph processing system (Expander) and deep learning framework (tensorflow) are separate systems. Spark looks to be built from the graph side (RDD) first and is now getting ML compo…

So..

MLLib seems awesome, but the devil is in the detail. Example that have burnt me include things like using LogisticRegression for classification only supports binary classification, the LibSVM support only support import, the GBT implantation is weak compared to eg XGBoost etc.

A lot of the time it is fine though.

Graph support.. hmm. GraphX is ok, but there are lots of things that eg NetworkX has the GraphX doesn't. In my experience, we've started a lot of projects with GraphX and abandoned them because GraphX's implementations didn't have the features we needed.

BTW, RDDs aren't graphs. I think you might be confusing the Spark directed-acyclic-graph (DAG) execution model with graph processing.

TensorFlow doesn't have as many general purpose ML algorithms. For example, I don't think there is a Random Forest in TF, and for 90% of ML problems RF is what you need.

But if you are doing Neural Network stuff then TF is exactly what you need.

Re: BigDL: Distributed Deep Learning on Apache Spark

#24

Post by sandGorgon »

nl wrote: post #23
sandGorgon wrote: post #19

Earlier quoted context omitted.

for someone just getting started on Spark - what do you mean "somewhat adequate" ? Because I see MLLib ( https://spark.apache.org/docs/2.0.2/mllib-guide.html ) and a quick glance shows me a lot of overlap with tensorflow. At google, their graph processing system (Expander) and deep learning framework (tensorflow) are separate systems. Spark looks to be built from the graph side (RDD) first and is now getting ML compo…

So.. MLLib seems awesome, but the devil is in the detail. Example that have burnt me include things like using LogisticRegression for classification only supports binary classification, the LibSVM support only support import, the GBT implantation is weak compared to eg XGBoost etc. A lot of the time it is fine though. Graph support.. hmm. GraphX is ok, but there are lots of things that eg NetworkX has the GraphX does…

thanks for that comment. Indeed we are looking at general purpose ML (gbm and logit regression being our primary usecases). I was not looking at RDD but rather Graphframes.

I see that you worked with GraphX and abandoned it. This is disappointing - we were really looking forward to Spark Graphframes with HBase as the oltp data store for graph data.

In your situation, how did you overcome the problems in Spark ? Did you use an accompanying toolkit to augment spark or did you build your own (hopefully not!).

Re: BigDL: Distributed Deep Learning on Apache Spark

#25

Post by Aeolos »

nl wrote: post #23
sandGorgon wrote: post #19

Earlier quoted context omitted.

for someone just getting started on Spark - what do you mean "somewhat adequate" ? Because I see MLLib ( https://spark.apache.org/docs/2.0.2/mllib-guide.html ) and a quick glance shows me a lot of overlap with tensorflow. At google, their graph processing system (Expander) and deep learning framework (tensorflow) are separate systems. Spark looks to be built from the graph side (RDD) first and is now getting ML compo…

So.. MLLib seems awesome, but the devil is in the detail. Example that have burnt me include things like using LogisticRegression for classification only supports binary classification, the LibSVM support only support import, the GBT implantation is weak compared to eg XGBoost etc. A lot of the time it is fine though. Graph support.. hmm. GraphX is ok, but there are lots of things that eg NetworkX has the GraphX does…

Tensorflow has a GPU-accelerated Random Forest implementation: https://github.com/tensorflow/tensorflow/blob/v0.10.0rc0/ten...

Re: BigDL: Distributed Deep Learning on Apache Spark

#26

Post by sandGorgon »

agibsonccc wrote: post #20
vonnik wrote: post #11

Deeplearning4j does this already. It has a huge community, a Scala API and does model import from Keras. It's important to note that Spark is not an efficient computation later -- it's best if used for fast ETL. If you get that wrong, your going to be training slow. https://deeplearning4j.org https://github.com/deeplearning4j/ScalNet https://deeplearning4j.org/model-import-keras https://gitter.im/deeplearning4j/deepl…

Chris my cofounder forgot to disclose he works on the project :). I"ll do it for him. I'd just like to say that as far as this niche is concerned. This is basically an attempt at "non gpus on spark". We are heavily biased towards cuda and distributed gpu applications: https://blogs.nvidia.com/blog/2016/10/06/how-skymind-nvidia-... I respect what intel is trying to do here, but it's going to take a lot more than "we b…

that sounds interesting.

so dl4j works with spark ? https://deeplearning4j.org/spark#how

is it because spark does "distributed computing" very efficiently ? In that case, would the apples-to-apples comparison be versus spark+tensorflow ? https://databricks.com/blog/2016/12/21/deep-learning-on-data...

Re: BigDL: Distributed Deep Learning on Apache Spark

#27

Post by nl »

Aeolos wrote: post #25
nl wrote: post #23

Earlier quoted context omitted.

So.. MLLib seems awesome, but the devil is in the detail. Example that have burnt me include things like using LogisticRegression for classification only supports binary classification, the LibSVM support only support import, the GBT implantation is weak compared to eg XGBoost etc. A lot of the time it is fine though. Graph support.. hmm. GraphX is ok, but there are lots of things that eg NetworkX has the GraphX does…

Tensorflow has a GPU-accelerated Random Forest implementation: https://github.com/tensorflow/tensorflow/blob/v0.10.0rc0/ten...

Nice. I'm glad to be wrong.

I'll point out that this is TF Contrib Learn, not TF Learn[1], or one of many other places where things might be implemented. Makes things a bit confusing.

[1] http://tflearn.org/

Re: BigDL: Distributed Deep Learning on Apache Spark

#28

Post by nl »

sandGorgon wrote: post #24
nl wrote: post #23

Earlier quoted context omitted.

So.. MLLib seems awesome, but the devil is in the detail. Example that have burnt me include things like using LogisticRegression for classification only supports binary classification, the LibSVM support only support import, the GBT implantation is weak compared to eg XGBoost etc. A lot of the time it is fine though. Graph support.. hmm. GraphX is ok, but there are lots of things that eg NetworkX has the GraphX does…

thanks for that comment. Indeed we are looking at general purpose ML (gbm and logit regression being our primary usecases). I was not looking at RDD but rather Graphframes. I see that you worked with GraphX and abandoned it. This is disappointing - we were really looking forward to Spark Graphframes with HBase as the oltp data store for graph data. In your situation, how did you overcome the problems in Spark ? Did y…

I think it's very hard to give general advice in this area. You are best off prototyping a deep spike into what you need, and seeing where things don't work.

What specific graph operations do you want?

If the stuff you need is there, then you might be fine! Note that the set of pre-built algorithms in GraphFrames is pretty small (https://graphframes.github.io/user-guide.html#graph-algorith...). It is pre-release though.

Graph stuff is generally hard, so I don't think there is a magic bullet here.

I mean, even just the Spark-using-HBase bit is non-trivial to do in a way that provides adequate performance. There are 3(?) different connectors, with pluses and minuses for each one. Making sure data locality is working will depend on you YARzn or Mesos setup, and debugging that is a nightmare.

In our case, we prefilter data in Spark then load into NetworkX. Works ok, mostly.

Re: BigDL: Distributed Deep Learning on Apache Spark

#29

Post by sandGorgon »

nl wrote: post #28
sandGorgon wrote: post #24

Earlier quoted context omitted.

thanks for that comment. Indeed we are looking at general purpose ML (gbm and logit regression being our primary usecases). I was not looking at RDD but rather Graphframes. I see that you worked with GraphX and abandoned it. This is disappointing - we were really looking forward to Spark Graphframes with HBase as the oltp data store for graph data. In your situation, how did you overcome the problems in Spark ? Did y…

I think it's very hard to give general advice in this area. You are best off prototyping a deep spike into what you need, and seeing where things don't work. What specific graph operations do you want? If the stuff you need is there, then you might be fine! Note that the set of pre-built algorithms in GraphFrames is pretty small ( https://graphframes.github.io/user-guide.html#graph-algorith... ). It is pre-release th…

well, it is not very different from what Google Expander does - https://research.googleblog.com/2016/10/graph-powered-machin...

our data sets have massively grown over the laat few months and now need a bigger solution. I think we will start off with a hosted solution like EMR - performance is not super critical right now (batch mode training)... but developer productivity is key.

Post reply on HN