Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dataflow/Beam and Spark: A Programming Model Comparison

cloud.google.com

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#11

Post by nda »

TheLogothete wrote: post #6

Hey, this is kind of offtopic, but figured still appropriate to ask; How come Google Cloud Storage can be used instead of HDFS? I'm comparing google/amazon/azure right now. Both Amazon and Azure have 2 types of storage options - the regular object storage (S3a and Blobs) and block storage (S3 and Data Lake Store). S3 and DLS can act as the file system for Hadoop themselves (meaning you can let the data sit there and…

GCS can be used instead of HDFS, Dataproc is shipped with GCS connector installed (https://github.com/GoogleCloudPlatform/bigdata-interop)

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#12

Post by TheLogothete »

nda wrote: post #11
TheLogothete wrote: post #6

Hey, this is kind of offtopic, but figured still appropriate to ask; How come Google Cloud Storage can be used instead of HDFS? I'm comparing google/amazon/azure right now. Both Amazon and Azure have 2 types of storage options - the regular object storage (S3a and Blobs) and block storage (S3 and Data Lake Store). S3 and DLS can act as the file system for Hadoop themselves (meaning you can let the data sit there and…

GCS can be used instead of HDFS, Dataproc is shipped with GCS connector installed ( https://github.com/GoogleCloudPlatform/bigdata-interop )

That... was kind of my point.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#13

Post by cbsmith »

EwanToo wrote: post #9
glogla wrote: post #5

Earlier quoted context omitted.

I haven't studied the code very carefully, but I think this is mostly PR piece. Not only using Spark from Java isn't a very good idea, but the "canonical" way to do transformations like that is using dataframes or datasets (which are from Spark 1.6 and provide some improvements over dataframes). Take it with a grain of salt.

I can't remember Google ever giving an open source project a kicking like it feels they've been giving Apache Spark recently, seems very unlike them.

It's less Google and more Google engineers, and from that standpoint it is totally normal fire them to kick the living daylights out of any code, particularly Google code.

Spark streaming really feels operationally immature compared to a lot of other stream processing frameworks, even Dataflow. The criticism is both unsurprising and warranted.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#14

Post by cbsmith »

kod wrote: post #8
nl wrote: post #4

I think that this is useful. One question though. Is there a reason you can't use the Spark Window functions? https://databricks.gitbooks.io/databricks-spark-reference-ap...

Spark windowing functions (like pretty much all notions of time in Spark streaming) are based on processing time, not event time.

So weird/annoying that it isn't at least event count, if not event time.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#15

Post by chimerasaurus »

TheLogothete wrote: post #6

Hey, this is kind of offtopic, but figured still appropriate to ask; How come Google Cloud Storage can be used instead of HDFS? I'm comparing google/amazon/azure right now. Both Amazon and Azure have 2 types of storage options - the regular object storage (S3a and Blobs) and block storage (S3 and Data Lake Store). S3 and DLS can act as the file system for Hadoop themselves (meaning you can let the data sit there and…

Since Cloud Dataproc is ephemeral it's going to be a (probably) better idea to use GCS over HDFS so there is no data loss.

Technically, Cloud Dataproc clusters have both HDFS (on PD) for write/read-intensive operations (and scratch space) along with the GCS connector. GCS is not the default file system, however.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#16

Post by glogla »

takidau wrote: post #10
glogla wrote: post #5

Earlier quoted context omitted.

I haven't studied the code very carefully, but I think this is mostly PR piece. Not only using Spark from Java isn't a very good idea, but the "canonical" way to do transformations like that is using dataframes or datasets (which are from Spark 1.6 and provide some improvements over dataframes). Take it with a grain of salt.

You're not going to get clean out-of-order processing semantics with any mode of Spark transformations. If you actually take the time to read the article, there's a section discussing the Java/Scala angle. The difference in code size is really secondary (though it is a difference). The difficulty in maintaining and evolving your pipeline over time using Spark is the main point, given the way important concepts become…

> If you actually take the time to read the article, there's a section discussing the Java/Scala angle.

They claim this isn't about the length of code, yet they select the most verbose way to use Spark and proudly display how long it is.

I mean sure, lack of event-time based processing is known limitation of Spark (and a pretty annoying one - though it is supposed to be worked on) but there are ways to write about it without code made to look bad on purpose.

EDIT: come to think of it, this whole article is "spark streaming can't do event time" written in thousands of words with contrived examples attached.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#17

Post by [deleted] »

glogla wrote: post #5
nl wrote: post #4

I think that this is useful. One question though. Is there a reason you can't use the Spark Window functions? https://databricks.gitbooks.io/databricks-spark-reference-ap...

I haven't studied the code very carefully, but I think this is mostly PR piece. Not only using Spark from Java isn't a very good idea, but the "canonical" way to do transformations like that is using dataframes or datasets (which are from Spark 1.6 and provide some improvements over dataframes). Take it with a grain of salt.

[deleted]

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#18

Post by teraflop »

I really like the Dataflow programming model, but this feels to me like an apples-to-oranges comparison.

Spark Streaming is a fully open-source project; although the Dataflow SDK is also OSS, my understanding is that the released version can only handle bounded datasets. Support for streaming (which is the major innovation, IMO) is only available in the form of stubs that call out to Google's paid, proprietary Dataflow service.

It's totally fine to compare an open-source project with a proprietary alternative, but I think it's odd that this article opens by talking about how the Dataflow SDK is being opened, and then spends all its time talking about proprietary features.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#19

Post by azurezyq »

teraflop wrote: post #18

I really like the Dataflow programming model, but this feels to me like an apples-to-oranges comparison. Spark Streaming is a fully open-source project; although the Dataflow SDK is also OSS, my understanding is that the released version can only handle bounded datasets. Support for streaming (which is the major innovation, IMO) is only available in the form of stubs that call out to Google's paid, proprietary Datafl…

Accoring to the proposal (https://wiki.apache.org/incubator/BeamProposal), OSS impl. of streaming is on the way, by Apache Flink, etc. The blog is just suggesting the model itself is superior, regardless of OSS or not.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#20

Post by teraflop »

azurezyq wrote: post #19
teraflop wrote: post #18

I really like the Dataflow programming model, but this feels to me like an apples-to-oranges comparison. Spark Streaming is a fully open-source project; although the Dataflow SDK is also OSS, my understanding is that the released version can only handle bounded datasets. Support for streaming (which is the major innovation, IMO) is only available in the form of stubs that call out to Google's paid, proprietary Datafl…

Accoring to the proposal ( https://wiki.apache.org/incubator/BeamProposal ), OSS impl. of streaming is on the way, by Apache Flink, etc. The blog is just suggesting the model itself is superior, regardless of OSS or not.

Huh, I can't find anything to that effect in the proposal. It does mention the existence of runners for Spark and Flink, but doesn't say they'll be getting streaming support. I had assumed it was unlikely for the same reason that this article talks about (lack of support for first-class processing by event time).

But assuming it's true. it's very welcome news! I'll be keeping a close eye on future releases.

Post reply on HN