A Scala API for Google Cloud Dataflow

It's a bit interesting that Cloudera went the opposite way than Spotify and fitted Google's Java API on top of Spark instead (so, changed the backend instead of the "frontend") [1].

[1] https://github.com/cloudera/spark-dataflow

Re: A Scala API for Google Cloud Dataflow

#7

Post by ecesena » Thu, Mar 24, 2016, 3:08 PM UTC

Any plan to port it to Beam?

Re: A Scala API for Google Cloud Dataflow

#8

Post by lucdurette » Thu, Mar 24, 2016, 4:32 PM UTC

Interesting project, glad to see more and more organisation are using Scala with Data projects.

Re: A Scala API for Google Cloud Dataflow

#9

Post by hnbroseph » Thu, Mar 24, 2016, 5:49 PM UTC

wiradikusuma wrote: post #2

scio is also the name for a portable molecular sensor: https://www.consumerphysics.com/myscio/scio

this is cool! thanks for mentioning it, i may have to grab myself a unit or two.

Re: A Scala API for Google Cloud Dataflow

#10

Post by sinisa » Thu, Mar 24, 2016, 8:22 PM UTC

samuell wrote: post #6

It's a bit interesting that Cloudera went the opposite way than Spotify and fitted Google's Java API on top of Spark instead (so, changed the backend instead of the "frontend") [1]. [1] https://github.com/cloudera/spark-dataflow

Scio author here.

A bit background: Spark and Flink are both frameworks with their own execution engine. Scalding is tightly coupled with Cascading + Hadoop as it's execution engine (also tez WIP). Dataflow Java SDK/Apache BEAM on the other hand is designed to be a simple abstraction with pluggable engines and Cloud Dataflow service is just one of the many runners possible.

Right now there are:

- local runner

- Dataflow runner, fully managed service in GCP

- Spark runner

- Flink runner

Scio wraps Dataflow Java SDK(Apache BEAM) and can potentially leverage any runner available.