Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Flink

github.com

Post reply on HN
Print view View original

21–30 of 65 posts

Re: Apache Flink

#21

Post by asavinov »

meritt wrote: post #4

Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data…

> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing.

Yes, conceptually they are very similar. If youu want something radically new then check out Bistro Streams: https://github.com/asavinov/bistro

Re: Apache Flink

#22

Post by marcyb5st »

meritt wrote: post #6
virgilp wrote: post #5

Earlier quoted context omitted.

> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. Well, no, you went too far. Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right) Spa…

On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident. I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework. > What's wrong with that? I believe it drastically reduces adoption of these tools because many of us avoid what appears to…

To be a bit more future-proof you should give Apache Beam a try. The same code should (theoretically) work with any of the supported runners[0] and so you could deploy it on top of the most suitable framework/technology for your specific workload. Moreover, at this point, the community has several examples of how to add an additional runner.

Edit: I didn't see you mentioned Beam in your second salvo :)

[0]: https://beam.apache.org/documentation/runners/capability-mat...

Re: Apache Flink

#23

Post by mxmxm »

meritt wrote: post #4

Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data…

Not quite. All of these are open-source projects but they are very different in many aspects:

* Apache Flink

Sophisticated stream processing framework with focus on robustness (managed memory) and correctness (exactly-once semantics)

* Apache Flume

Tailored towards log data.

* Apache Storm

First stream processing framework. Legacy.

* Apache Samza

Only used at LinkedIn. Tight to Hadoop's YARN.

* Apache Spark

Only great in batch processing.

* Apache Apex

Dead project. Tight to Hadoop's YARN.

* Apache Kafka

A distributed message queue with simple stream processing built on top via the Confluent Platform.

Re: Apache Flink

#24

Post by dankohn1 »

mxmxm wrote: post #23
meritt wrote: post #4

Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data…

Not quite. All of these are open-source projects but they are very different in many aspects: * Apache Flink Sophisticated stream processing framework with focus on robustness (managed memory) and correctness (exactly-once semantics) * Apache Flume Tailored towards log data. * Apache Storm First stream processing framework. Legacy. * Apache Samza Only used at LinkedIn. Tight to Hadoop's YARN. * Apache Spark Only grea…

Nice list. There are also non-Apache projects like NATS, hosted by CNCF.

Here are the 19 streaming and messaging projects and products that CNCF is tracking: https://landscape.cncf.io/category=streaming-messaging&forma...

Re: Apache Flink

#25

Post by StevePerkins »

meritt wrote: post #6
virgilp wrote: post #5

Earlier quoted context omitted.

> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. Well, no, you went too far. Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right) Spa…

On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident. I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework. > What's wrong with that? I believe it drastically reduces adoption of these tools because many of us avoid what appears to…

In the case of Spark and Flink, I wouldn't say that batch processing versus realtime stream processing are "subtle differences". That's akin to arguing that relational databases vs. document stores vs. timeseries databases just "muddy the waters".

Hacker News and Reddit have a lot of interesting discussion. But the audience skews toward client-side webdev, and students or younger developers. An audience accustomed to libraries and frameworks that you can reason about with fairly low learning curve, and spin up in a Codepen to see visually right away.

Heavy-lifting server side tools, especially those who only earn their keep at scale, are a different beast. And that's OKAY. Quite frankly, if you're "not sure" whether you need a stream processing platform in your architecture, then YOU DON'T. Aside from some consultants and salespeople, no one's really going to push you toward adoption of this stuff.

In the overwhelming majority of use cases, what you need is a tiny microservice (in your language of choice). Which reads from a Kafka or Rabbit topic, and stores state in your cache system of choice. By the time you reach the scale where that's not suitable, your organization probably won't need a web forum thread to educate you on what the vendor landscape looks like.

Re: Apache Flink

#26

Post by humanbeinc »

11sdmb8m wrote: post #17
IshKebab wrote: post #16

Earlier quoted context omitted.

I wish people would stop making this predictable and tedious comment that adds nothing to the conversation.

"Make a PR" actually means "This seems not to be relevant enough for the current maintainers to do it, and if it feels important to you, you should write it yourself and post a patch", which is a good answer.

No, it's not and you know it

Re: Apache Flink

#27

Post by vageli »

11sdmb8m wrote: post #17
IshKebab wrote: post #16

Earlier quoted context omitted.

I wish people would stop making this predictable and tedious comment that adds nothing to the conversation.

"Make a PR" actually means "This seems not to be relevant enough for the current maintainers to do it, and if it feels important to you, you should write it yourself and post a patch", which is a good answer.

> "Make a PR" actually means "This seems not to be relevant enough for the current maintainers to do it, and if it feels important to you, you should write it yourself and post a patch", which is a good answer.

So we just assume that a desire to do something translates to an _ability_ to do it?

Re: Apache Flink

#28

Post by marcinzm »

rakoo wrote: post #9
virgilp wrote: post #7

Earlier quoted context omitted.

I said "you went to far" to claim Kafka, Spark and Flink do basically the same thing. It's reasonable to use all 3 of them in the same team - so they clearly don't do the same thing. > I believe it drastically reduces adoption of these tools As does any competition. Tons of smartphone makers = less adoption for any one of them - and many will close down. Still not a bad thing.

> As does any competition. Those are open-source frameworks, which is antithetical to competition. The whole point of open-source is that the contributions of one entity can benefit everyone, while in this situation, having so many frameworks "dilutes" the efforts because the same problems need to be solved in each framework. Smartphone makers aren't here to share their technical advances with everyone, they're here…

>I'm not saying we should purposefully agree on killing all but one framework, but the ideal situation is if one or two come out at the top as being just the best in their category so that everyone just use them, and all efforts converge towards those.

That assumes you can have a single framework with all the features people want without causing issues (conflicting basic designs, high cost to maintain, deployment costs, configuration costs, etc.).

Re: Apache Flink

#29

Post by michaelmior »

mxmxm wrote: post #23
meritt wrote: post #4

Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data…

Not quite. All of these are open-source projects but they are very different in many aspects: * Apache Flink Sophisticated stream processing framework with focus on robustness (managed memory) and correctness (exactly-once semantics) * Apache Flume Tailored towards log data. * Apache Storm First stream processing framework. Legacy. * Apache Samza Only used at LinkedIn. Tight to Hadoop's YARN. * Apache Spark Only grea…

> Only used at LinkedIn.

And Intuit, Uber, Netflix, VMWare, ....

https://cwiki.apache.org/confluence/display/SAMZA/Powered+By

Post reply on HN