Dataflow/Beam and Spark: A Programming Model Comparison

#1

Post by vgt » Wed, Feb 03, 2016, 7:34 PM UTC

Dataflow/Beam and Spark: A Programming Model Comparison
cloud.google.com

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#2

Post by dang » Wed, Feb 03, 2016, 8:31 PM UTC

Url changed from https://cloud.google.com/blog/big-data/2016/02/comparing-the..., which points to this.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#3

Post by vgt » Wed, Feb 03, 2016, 8:35 PM UTC

dang wrote: post #2

Url changed from https://cloud.google.com/blog/big-data/2016/02/comparing-the... , which points to this.

Thanks! The original URL is a new developer-focused Big Data blog by Google and Google customers:

https://cloud.google.com/blog/big-data/

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#4

Post by nl » Wed, Feb 03, 2016, 9:09 PM UTC

I think that this is useful.

One question though. Is there a reason you can't use the Spark Window functions? https://databricks.gitbooks.io/databricks-spark-reference-ap...

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#5

Post by glogla » Wed, Feb 03, 2016, 9:16 PM UTC

nl wrote: post #4

I think that this is useful. One question though. Is there a reason you can't use the Spark Window functions? https://databricks.gitbooks.io/databricks-spark-reference-ap...

I haven't studied the code very carefully, but I think this is mostly PR piece. Not only using Spark from Java isn't a very good idea, but the "canonical" way to do transformations like that is using dataframes or datasets (which are from Spark 1.6 and provide some improvements over dataframes).

Take it with a grain of salt.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#6

Post by TheLogothete » Wed, Feb 03, 2016, 9:40 PM UTC

Hey, this is kind of offtopic, but figured still appropriate to ask;

How come Google Cloud Storage can be used instead of HDFS? I'm comparing google/amazon/azure right now. Both Amazon and Azure have 2 types of storage options - the regular object storage (S3a and Blobs) and block storage (S3 and Data Lake Store). S3 and DLS can act as the file system for Hadoop themselves (meaning you can let the data sit there and fire up clusters just for processing when needed), but they cannot interface with tools like the regular storage.

Meanwhile, Google's storage is like regular object storage, but you can run map/reduce (dataproc) and Spark on it.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#7

Post by continuational » Wed, Feb 03, 2016, 9:41 PM UTC

glogla wrote: post #5

nl wrote: post #4

I think that this is useful. One question though. Is there a reason you can't use the Spark Window functions? https://databricks.gitbooks.io/databricks-spark-reference-ap...

I haven't studied the code very carefully, but I think this is mostly PR piece. Not only using Spark from Java isn't a very good idea, but the "canonical" way to do transformations like that is using dataframes or datasets (which are from Spark 1.6 and provide some improvements over dataframes). Take it with a grain of salt.

When there's an obvious misrepresentation of the competition like in this piece, it's a big red flag for me. I think it hurts PR in the end.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#8

Post by kod » Wed, Feb 03, 2016, 9:56 PM UTC

nl wrote: post #4

I think that this is useful. One question though. Is there a reason you can't use the Spark Window functions? https://databricks.gitbooks.io/databricks-spark-reference-ap...

Spark windowing functions (like pretty much all notions of time in Spark streaming) are based on processing time, not event time.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#9

Post by EwanToo » Wed, Feb 03, 2016, 9:57 PM UTC

glogla wrote: post #5

nl wrote: post #4

I think that this is useful. One question though. Is there a reason you can't use the Spark Window functions? https://databricks.gitbooks.io/databricks-spark-reference-ap...

I haven't studied the code very carefully, but I think this is mostly PR piece. Not only using Spark from Java isn't a very good idea, but the "canonical" way to do transformations like that is using dataframes or datasets (which are from Spark 1.6 and provide some improvements over dataframes). Take it with a grain of salt.

I can't remember Google ever giving an open source project a kicking like it feels they've been giving Apache Spark recently, seems very unlike them.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#10

Post by takidau » Wed, Feb 03, 2016, 10:03 PM UTC

glogla wrote: post #5

nl wrote: post #4

I think that this is useful. One question though. Is there a reason you can't use the Spark Window functions? https://databricks.gitbooks.io/databricks-spark-reference-ap...

I haven't studied the code very carefully, but I think this is mostly PR piece. Not only using Spark from Java isn't a very good idea, but the "canonical" way to do transformations like that is using dataframes or datasets (which are from Spark 1.6 and provide some improvements over dataframes). Take it with a grain of salt.

You're not going to get clean out-of-order processing semantics with any mode of Spark transformations. If you actually take the time to read the article, there's a section discussing the Java/Scala angle. The difference in code size is really secondary (though it is a difference). The difficulty in maintaining and evolving your pipeline over time using Spark is the main point, given the way important concepts become conflated with their API (any version of it). This all comes across much more clearly for those that actually take the time to read the words.