Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Flink

github.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Apache Flink

#31

Post by marcinzm »

meritt wrote: post #6
virgilp wrote: post #5

Earlier quoted context omitted.

> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. Well, no, you went too far. Kafka is primarily used for communication & data transport, by most people (can be used in other ways, and it has the KafkaStreams library that enables you do to some computation on said data - but it is, primarily, a transport & communication mechanism; also maybe storage, if you squint right) Spa…

On the contrary, I didn't go far enough. I didn't talk about Apache Gearpump, NiFi, Beam, Ignite, or Trident. I know there are subtle differences in each specific technology, and that's probably the same justifications used to support the thesis when building yet another very similar framework. > What's wrong with that? I believe it drastically reduces adoption of these tools because many of us avoid what appears to…

>I believe it drastically reduces adoption of these tools because many of us avoid what appears to be bandwagon technologies as we don't want to consciously add layers of future technical debt when a majority of these projects will be abandoned.

Doesn't seem to be impacting adoption of the major ones from what I can see (Spark, Kafka, Flink). Assuming you're at a scale where you need these technologies. Which most companies are not so it's a good thing they don't implement them just for kicks or as "future proofing." If you have a business problem that the main frameworks can't solve then trying out some of the other ones may be worth the cost. There's also ongoing support for older frameworks (Storm) so it's not like your code becomes useless.

The approach open source is taking to resolve this issue is to embrace the diversity but create unified APIs on top of it. Apache Beam and Arrow for defining data workflows and data exchange formats respectively. There's also always SQL which works with a lot of the more data warehouse solutions out there (with some tweaks per solution unfortunately).

Re: Apache Flink

#32

Post by lmm »

vageli wrote: post #27
11sdmb8m wrote: post #17

Earlier quoted context omitted.

"Make a PR" actually means "This seems not to be relevant enough for the current maintainers to do it, and if it feels important to you, you should write it yourself and post a patch", which is a good answer.

> "Make a PR" actually means "This seems not to be relevant enough for the current maintainers to do it, and if it feels important to you, you should write it yourself and post a patch", which is a good answer. So we just assume that a desire to do something translates to an _ability_ to do it?

If you want it done and don't have the skills you can hire someone who does.

Re: Apache Flink

#34

Post by buboard »

meritt wrote: post #4

Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data…

Solving problems of the internet behemoths -> creating opportunity to be acquired.

Re: Apache Flink

#35

Post by ataturk »

rakoo wrote: post #9
virgilp wrote: post #7

Earlier quoted context omitted.

I said "you went to far" to claim Kafka, Spark and Flink do basically the same thing. It's reasonable to use all 3 of them in the same team - so they clearly don't do the same thing. > I believe it drastically reduces adoption of these tools As does any competition. Tons of smartphone makers = less adoption for any one of them - and many will close down. Still not a bad thing.

> As does any competition. Those are open-source frameworks, which is antithetical to competition. The whole point of open-source is that the contributions of one entity can benefit everyone, while in this situation, having so many frameworks "dilutes" the efforts because the same problems need to be solved in each framework. Smartphone makers aren't here to share their technical advances with everyone, they're here…

If you've been around awhile no doubt you will recall the web application server wars (Spring, J2EE, Tapestry, Struts, etc.) and the JVM language wars (Java, Scala, Groovy, Clojure) and now I guess we're in the Big Data wars...

Re: Apache Flink

#36

Post by rsynnott »

_ZeD_ wrote: post #8

"Prerequisites for building Flink: Unix-like environment (we use Linux, Mac OS X, Cygwin) Java 8 (Java 9 and 10 are not yet supported)" Sigh is too much to ask for proper crossplatform support? And when the hell they will add support for recent versions of Java????

Pretty much all of these Hadoop-adjacent things are more or less Linux only, and certainly unix-y-thing-only. What other platform do you want to run it on?

Re: Apache Flink

#37

Post by rsynnott »

meritt wrote: post #4

Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data…

In the beginning, there was Hadoop, which was just a MapReduce clone, more or less.

And then there was Hadoop 2, which was kind of meant to do everything.

And that didn't really work out, so there was specialisation.

I don't really see the problem. Many of these do different things, others do similar things different ways. Some are essentially dead ends (I doubt there are many new users of Storm, say). This all seems pretty normal.

Re: Apache Flink

#38

Post by lenticular »

Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't need them. I see it all the time with ML work. You wind up using a cluster to overcome the memory inefficiency of Spark, when you could have just used a single machine. For example, I've done huge graph clustering models on a single machine just by being smart about memory consumption. It would have taken an enormous and expensive Spark cluster.

Re: Apache Flink

#39

Post by manigandham »

lenticular wrote: post #38

Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't need them. I see it all the time with ML work. You wind up using a cluster to overcome the memory inefficiency of Spark, when you could have just used a single machine. For example, I've done huge graph clustering models on a single machine just by being smar…

We have also seen this. The huge memory capacities now offered in the cloud makes single node processing very capable with entire TB datasets fitting into RAM, and running quickly enough to offset the hourly cost. Spark clusters are more efficient though for longer-running or continuous background processes.

Re: Apache Flink

#40

Post by manigandham »

PurpleRamen wrote: post #18
rakoo wrote: post #9

Earlier quoted context omitted.

> As does any competition. Those are open-source frameworks, which is antithetical to competition. The whole point of open-source is that the contributions of one entity can benefit everyone, while in this situation, having so many frameworks "dilutes" the efforts because the same problems need to be solved in each framework. Smartphone makers aren't here to share their technical advances with everyone, they're here…

Since when is there no competition and diversity in open source? Isn't too much choice usually a major complain with linux, programming languages, frameworks, etc?

Nobody said that. But having too many competing overlapping products does mean more fractured effort spread amongst all of them.
Post reply on HN