Apache Flink
github.com
Apache Flink
1–10 of 65 posts
Re: Apache Flink
#2Does anyone have experience running in production?
If you have an article about using it or slides please send links.
Re: Apache Flink
#3Does anyone have experience running in production? If you have an article about using it or slides please send links.
Not sure why the Flink github site got linked. Here's a list of companies using it in production, with links to details.
Re: Apache Flink
#4Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there.
* Apache Flink is an open source stream processing framework
* Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data.
* Apache Storm is a distributed stream processing computation framework
* Apache Samza is an open-source near-realtime, asynchronous computational framework for stream processing
* Apache Spark is an open-source distributed general-purpose cluster-computing framework.
* Apache Apex is a YARN-native platform that unifies stream and batch processing.
* Apache Kafka is an open-source stream-processing software platform
Re: Apache Flink
#5Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data…
> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing.
Well, no, you went too far.
Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right)
Spark and Flink might be similar on first sight, but if you look a bit closer you realize Spark is primarily geared towards batch workloads, and Flink towards realtime. Sure, you can do micro-batch in Spark and pretend that's realtime stream processing, but the focus of it is fairly clear - as is the focus of Flink. So both have legitimate rights to exist.
I'm not sure about the others, haven't used them. There may be indeed consistent overlap - but I'm sure they are different approaches. What's wrong with that?
Re: Apache Flink
#6Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data…
> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. Well, no, you went too far. Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right) Spa…
On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident.
I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework.
> What's wrong with that?
I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.
Re: Apache Flink
#7Earlier quoted context omitted.
> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. Well, no, you went too far. Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right) Spa…
On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident. I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework. > What's wrong with that? I believe it drastically reduces adoption of these tools because many of us avoid what appears to…
I said "you went to far" to claim Kafka, Spark and Flink do basically the same thing. It's reasonable to use all 3 of them in the same team - so they clearly don't do the same thing.
> I believe it drastically reduces adoption of these tools
As does any competition. Tons of smartphone makers = less adoption for any one of them - and many will close down. Still not a bad thing.
Re: Apache Flink
#8"Prerequisites for building Flink:
Unix-like environment (we use Linux, Mac OS X, Cygwin)
Java 8 (Java 9 and 10 are not yet supported)"
Sigh is too much to ask for proper crossplatform support? And when the hell they will add support for recent versions of Java????
Re: Apache Flink
#9Earlier quoted context omitted.
On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident. I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework. > What's wrong with that? I believe it drastically reduces adoption of these tools because many of us avoid what appears to…
I said "you went to far" to claim Kafka, Spark and Flink do basically the same thing. It's reasonable to use all 3 of them in the same team - so they clearly don't do the same thing. > I believe it drastically reduces adoption of these tools As does any competition. Tons of smartphone makers = less adoption for any one of them - and many will close down. Still not a bad thing.
> As does any competition.
Those are open-source frameworks, which is antithetical to competition. The whole point of open-source is that the contributions of one entity can benefit everyone, while in this situation, having so many frameworks "dilutes" the efforts because the same problems need to be solved in each framework. Smartphone makers aren't here to share their technical advances with everyone, they're here to make money.
I'm not saying we should purposefully agree on killing all but one framework, but the ideal situation is if one or two come out at the top as being just the best in their category so that everyone just use them, and all efforts converge towards those.
Re: Apache Flink
#10"Prerequisites for building Flink: Unix-like environment (we use Linux, Mac OS X, Cygwin) Java 8 (Java 9 and 10 are not yet supported)" Sigh is too much to ask for proper crossplatform support? And when the hell they will add support for recent versions of Java????
Make a PR