Dataflow/Beam and Spark: A Programming Model Comparison • Hacker News Board

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#31

Post by andrioni » Thu, Feb 04, 2016, 10:27 AM UTC

TheLogothete wrote: post #6

Hey, this is kind of offtopic, but figured still appropriate to ask; How come Google Cloud Storage can be used instead of HDFS? I'm comparing google/amazon/azure right now. Both Amazon and Azure have 2 types of storage options - the regular object storage (S3a and Blobs) and block storage (S3 and Data Lake Store). S3 and DLS can act as the file system for Hadoop themselves (meaning you can let the data sit there and…

Most people (in my experience) don't use the old S3 block filesystem protocol anymore, and just use object storage instead (with either S3n, S3a or the proprietary EMRFS).

The Hadoop FileSystem interface doesn't really force an specific underlying implementation, and you can even use "local" filesystems without any issue, in fact, IIRC, MapR-FS is just an optimized NFS drive, i.e. a shared network drive.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#32

Post by samuell » Thu, Feb 04, 2016, 11:56 AM UTC

lern_too_spel wrote: post #29

samuell wrote: post #27

I'm much disturbed that the big G hijacked the dataflow term, to suddenly mean their specific - rather involved I must say - dataflow based programming model . The real dataflow [1] is a much broader term that doesn't outline specifics like programming semantics. Seems as if they're trying to ride the wave of the recent upsurge in interest in dataflow in general (with sub-fields such as Flow-based programming, and im…

It's called Google Cloud Dataflow. Your complaint is like saying Google Cloud Platform is hijacking the meaning of Platform.

Yes, but if you look at the use of the word in the linked post, you will see that plain "Dataflow" is almost exclusively used - even in the title.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#33

Post by cbsmith » Fri, Feb 05, 2016, 2:40 AM UTC

neeleshs wrote: post #28

cbsmith wrote: post #13

Earlier quoted context omitted.

It's less Google and more Google engineers, and from that standpoint it is totally normal fire them to kick the living daylights out of any code, particularly Google code. Spark streaming really feels operationally immature compared to a lot of other stream processing frameworks, even Dataflow. The criticism is both unsurprising and warranted.

What other stream processing frameworks do you prefer in the place of spark streaming?Storm is pretty mature, but does not play very well with YARN last I tried. Flink looks pretty good, but is fairly new. Samza is another one. I'm curious to know if you have any specific issues with spark streaming's operational immaturity. Some things I don't like in general are: # Backpressure algo is fairly new, but pluggable # D…

Storm has the community around it, though it shows signs of decline (perhaps the release of YARN-friendly Heron will help). There are a lot of suitors for developer affection. Fink is relatively green. Samza looks good, but is still picking up momentum in the community. DataTorrent seems to have some moment with its Apache Apex.

Regardless, most of these stream processing frameworks are still very much in early days and lack a lot of the sophistication you find in custom in house systems (such as found at... Google ;-). The open source world will no doubt catch up and overtake those systems, but right now there is still enough of a gap that it is rather painful.

Spark streaming pains:

1. Backpressure & stragglers. Duh. 2. Setup & tear down is still rough, even compared to Storm. 3. The whole context singleton thing means you need a new VM for each job, which annoys the #@$@#$ out of me. 4. Error handling isn't just unclear, it's kind of disastrous. 5. You can feel its "batch" heritage in lots of places, not just the stragglers. For some that is a feature, for me, a bug, even though with Storm I use Trident. 6. When a job runs amuck, it's a pain to recover from it. Storm is no picnic either, but it is indeed better.

Re: Dataflow/Beam and Spark: A Programming Model Comparison

#34

Post by jkff » Fri, Feb 05, 2016, 4:05 PM UTC

andrioni wrote: post #30

takidau wrote: post #10

Earlier quoted context omitted.

You're not going to get clean out-of-order processing semantics with any mode of Spark transformations. If you actually take the time to read the article, there's a section discussing the Java/Scala angle. The difference in code size is really secondary (though it is a difference). The difficulty in maintaining and evolving your pipeline over time using Spark is the main point, given the way important concepts become…

What do you think about Cloudera's Spark backend to Google Cloud Dataflow pipelines[1]? [1]: https://github.com/cloudera/spark-dataflow

Hi, I'm also an engineer on the Dataflow team. This is a very exciting project! However it is still constrained by Spark's limitations outlined in this post in terms of what Dataflow pipelines it can run. For example, currently it doesn't support session windows, it windows based on processing time instead of event time, and it doesn't seem to support triggers (the word "trigger" doesn't appear in the code base). AFAIK Spark has plans to add support for event time, and then it will be possible to make this runner run more pipelines consistently with the semantics of the Dataflow model. It is also likely that the capabilities of this runner will expand greatly as https://wiki.apache.org/incubator/BeamProposal progresses.