Apache Flink 1.9.0 Release Announcement

continuations wrote: post #3

Apache has a large number of stream processing frameworks: Flink vs Spark vs Storm vs Kafka vs Samza vs Apex How do they compare? How would you choose which one to use?

The only two of those I know are Kafka and Flink. For those two: Flink is much more full-featured and performant (basically the full Google DataFlow API, and several orders of magnitude faster than Kafka Streaming), but Kafka Streaming has a stupid simple API that is useful if you need streaming because $reason but don't care about scaling up to infinity. If you're doing some really hacky demoware, Kafka Streaming will probably be faster to spin up because you just need the Kafka Streaming jar and a Kafka cluster.

Re: Apache Flink 1.9.0 Release Announcement

#6

Post by barrkel » Sun, Aug 25, 2019, 2:00 AM UTC

continuations wrote: post #3

Apache has a large number of stream processing frameworks: Flink vs Spark vs Storm vs Kafka vs Samza vs Apex How do they compare? How would you choose which one to use?

You don't need me to search the internet for you, but https://thenewstack.io/apache-gets-another-real-time-stream-... has some comparisons.

There's also Apache Beam, which is an API for streaming, and has Flink and Apex execution engines. Google's Cloud Dataflow is another implementation of Apache Beam.

As to which one to choose, you need to evaluate them, there's no simple answers. If you have Hadoop already then Apex may be a better fit than Flink; OTOH if you do Akka stuff already, then Flink might integrate better with your stack. If you have more batch than streaming use cases, maybe you want Spark. Etc.

Re: Apache Flink 1.9.0 Release Announcement

#7

Post by StevePerkins » Sun, Aug 25, 2019, 2:05 AM UTC

continuations wrote: post #3

Apache has a large number of stream processing frameworks: Flink vs Spark vs Storm vs Kafka vs Samza vs Apex How do they compare? How would you choose which one to use?

I don't have experience with Samza or Apex, but as for the first three:

1. Flink - Focused on stateful stream processing.

2. Spark - Focused on batch processing. Can be used for continuous streams, but approaches them as "micro-batches".

3. Kafka - A message queue system (for all practical purposes). Has an optional stream processing add-on for basic needs.

Separate use cases and strengths aside, it's worth calling out that all of these products are primarily backed by completely different companies. Apache is a consortium made of many companies, and serves as common branding for "community editions" of their "enterprise edition" products. There can quite a lot of overlap between sponsored products in this consortium.

Re: Apache Flink 1.9.0 Release Announcement

#8

Post by lern_too_spel » Sun, Aug 25, 2019, 3:03 AM UTC

StevePerkins wrote: post #7

continuations wrote: post #3

Apache has a large number of stream processing frameworks: Flink vs Spark vs Storm vs Kafka vs Samza vs Apex How do they compare? How would you choose which one to use?

I don't have experience with Samza or Apex, but as for the first three: 1. Flink - Focused on stateful stream processing. 2. Spark - Focused on batch processing. Can be used for continuous streams, but approaches them as "micro-batches". 3. Kafka - A message queue system (for all practical purposes). Has an optional stream processing add-on for basic needs. Separate use cases and strengths aside, it's worth calling o…

Spark supports both microbatch and continuous stream processing.

Apache Software Foundation is not a consortium made of many companies but a single non-profit that provides organizational support for open source projects, some of which have contributors employed as such by other companies and some of which have only volunteer contributors.

Re: Apache Flink 1.9.0 Release Announcement

#9

Post by nivertech » Sun, Aug 25, 2019, 5:00 AM UTC

StevePerkins wrote: post #7

continuations wrote: post #3

Apache has a large number of stream processing frameworks: Flink vs Spark vs Storm vs Kafka vs Samza vs Apex How do they compare? How would you choose which one to use?

I don't have experience with Samza or Apex, but as for the first three: 1. Flink - Focused on stateful stream processing. 2. Spark - Focused on batch processing. Can be used for continuous streams, but approaches them as "micro-batches". 3. Kafka - A message queue system (for all practical purposes). Has an optional stream processing add-on for basic needs. Separate use cases and strengths aside, it's worth calling o…

4. Apache BEAM (same model as Google Cloud Dataflow)

Re: Apache Flink 1.9.0 Release Announcement

#10

Post by BFLpL0QNek » Sun, Aug 25, 2019, 6:20 AM UTC

jdm2212 wrote: post #5

continuations wrote: post #3

Apache has a large number of stream processing frameworks: Flink vs Spark vs Storm vs Kafka vs Samza vs Apex How do they compare? How would you choose which one to use?

The only two of those I know are Kafka and Flink. For those two: Flink is much more full-featured and performant (basically the full Google DataFlow API, and several orders of magnitude faster than Kafka Streaming), but Kafka Streaming has a stupid simple API that is useful if you need streaming because $reason but don't care about scaling up to infinity. If you're doing some really hacky demoware, Kafka Streaming wi…

Do you have any numbers to back up Flink is faster than KStreams, also under what scenario?

I am genuinely interested as use KStreams a lot but the engineering discipline in the API leads a lot to be desired and more than happy to switch the API if Flink is that much better.