Deep Learning with Spark and TensorFlow
databricks.com
Deep Learning with Spark and TensorFlow
1–10 of 31 posts
Re: Deep Learning with Spark and TensorFlow
#2Impressive, but it seems an inversion of paradigms. Small data to compute ratios is usually associated with high performance computing (HPC). Why use Spark when the data is small and is broadcast to each worker? You have to pay the serialization-deserialization penalties of moving the data from Python to JVM and back again. In fact the JVM isn't really needed here at all since all the computation is done in the pure-Python workers in an embarrassingly-parallel way. Seems to me that you would just move onto an HPC and use TensorFlow within a IPython.parallel paradigm and be done much sooner.
Re: Deep Learning with Spark and TensorFlow
#3Impressive, but it seems an inversion of paradigms. Small data to compute ratios is usually associated with high performance computing (HPC). Why use Spark when the data is small and is broadcast to each worker? You have to pay the serialization-deserialization penalties of moving the data from Python to JVM and back again. In fact the JVM isn't really needed here at all since all the computation is done in the pure-…
The "broadcast" is pretty cheap because often you already have the data in some distributed file system, or if on a single node the network bandwidth is pretty high. The problem with a lot of the deep learning workloads is that it is very compute intensive and as a result takes a long time to run. For example, it is not uncommon to take a week to train some models.
Re: Deep Learning with Spark and TensorFlow
#4That article reminded me of this: http://i.imgur.com/XQJ3ACO.jpg
Re: Deep Learning with Spark and TensorFlow
#5Impressive, but it seems an inversion of paradigms. Small data to compute ratios is usually associated with high performance computing (HPC). Why use Spark when the data is small and is broadcast to each worker? You have to pay the serialization-deserialization penalties of moving the data from Python to JVM and back again. In fact the JVM isn't really needed here at all since all the computation is done in the pure-…
The "broadcast" is pretty cheap because often you already have the data in some distributed file system, or if on a single node the network bandwidth is pretty high. The problem with a lot of the deep learning workloads is that it is very compute intensive and as a result takes a long time to run. For example, it is not uncommon to take a week to train some models.
Deep learning workloads are typically compute-intensive, but they also tend to be extremely I/O intensive, and convergence may depend on a synchronous step where all the nodes must finish making their contribution to the model before any of them can continue. (This may not be quite true though -- see Google's DistBelief paper--but most frameworks work this way). Often times, adding more machines to a cluster may make training proportionally slower.
Re: Deep Learning with Spark and TensorFlow
#6Earlier quoted context omitted.
The "broadcast" is pretty cheap because often you already have the data in some distributed file system, or if on a single node the network bandwidth is pretty high. The problem with a lot of the deep learning workloads is that it is very compute intensive and as a result takes a long time to run. For example, it is not uncommon to take a week to train some models.
Deep learning workloads are typically compute-intensive, but they also tend to be extremely I/O intensive, and convergence may depend on a synchronous step where all the nodes must finish making their contribution to the model before any of them can continue. (This may not be quite true though -- see Google's DistBelief paper--but most frameworks work this way). Often times, adding more machines to a cluster may make…
Did you actually read the article? It was using Spark to parallelize hyperparameter tuning, which is embarrassingly parallel.
Re: Deep Learning with Spark and TensorFlow
#7That article reminded me of this: http://i.imgur.com/XQJ3ACO.jpg
The blog post actually provides code to reproduce all the steps and the chart. See
http://go.databricks.com/hubfs/notebooks/TensorFlow/Distribu...
http://go.databricks.com/hubfs/notebooks/TensorFlow/Test_dis...
You might've missed the section "How do I use it?" Maybe we should've made that section more obvious.
Re: Deep Learning with Spark and TensorFlow
#8That article reminded me of this: http://i.imgur.com/XQJ3ACO.jpg
I am still laughing from that graphic. So simple, but, you know what? So true, too.
Re: Deep Learning with Spark and TensorFlow
#9Earlier quoted context omitted.
Deep learning workloads are typically compute-intensive, but they also tend to be extremely I/O intensive, and convergence may depend on a synchronous step where all the nodes must finish making their contribution to the model before any of them can continue. (This may not be quite true though -- see Google's DistBelief paper--but most frameworks work this way). Often times, adding more machines to a cluster may make…
Did you actually read the article? It was using Spark to parallelize hyperparameter tuning, which is embarrassingly parallel.
Why not just use GNU Parallel (or something similar) instead of Spark?
Re: Deep Learning with Spark and TensorFlow
#10Earlier quoted context omitted.
Did you actually read the article? It was using Spark to parallelize hyperparameter tuning, which is embarrassingly parallel.
Why not just use GNU Parallel (or something similar) instead of Spark?
I think this could have been done with GNU parallel. One advantage I see with Spark is that is that it is easier to interact with Python, for example these two lines are all is needed to call the relevant Python function:
urls = sc.parallelize(batched_data)
labelled_images = urls.flatMap(apply_batch)