Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Flink

github.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Apache Flink

#41

Post by diehunde »

meritt wrote: post #4

Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data…

> Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing.

That's like saying C++, Java, C#, Scala, Python, Clojure... all do basically the same thing

Re: Apache Flink

#42

Post by justicezyx »

rsynnott wrote: post #37
meritt wrote: post #4

Apache Flink, Flume, Storm, Samza, Spark, Apex, and Kafka all do basically the same thing. I feel like this is a bit overboard. And this is before we talk about the non-Apache stream-processing frameworks out there. * Apache Flink is an open source stream processing framework * Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggregating, and moving large amounts of log data…

In the beginning, there was Hadoop, which was just a MapReduce clone, more or less. And then there was Hadoop 2, which was kind of meant to do everything. And that didn't really work out, so there was specialisation. I don't really see the problem. Many of these do different things, others do similar things different ways. Some are essentially dead ends (I doubt there are many new users of Storm, say). This all seems…

No Hadoop does more than MapReduce, much more. Primarily: MapReduce does not have a distributed file-like block storage.

Re: Apache Flink

#43

Post by bunderbunder »

lenticular wrote: post #38

Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't need them. I see it all the time with ML work. You wind up using a cluster to overcome the memory inefficiency of Spark, when you could have just used a single machine. For example, I've done huge graph clustering models on a single machine just by being smar…

This has been my experience, too. I worked at just one place that had a really good handle on high-volume, high-velocity streaming data, and they didn't use Flink or Storm or Kafka or anything like that. They mostly just used the KISS principle and a protobuf-style wire format.[1]

There is definitely a point where these sorts of scale-out-centric solutions are unavoidable. Short of that point, though, they're probably best avoided.

[1]: (It's truly amazing how many CPU cycles you can reclaim just by removing branch instructions from your message deserialization code.)

Re: Apache Flink

#44

Post by bunderbunder »

justicezyx wrote: post #42
rsynnott wrote: post #37

Earlier quoted context omitted.

In the beginning, there was Hadoop, which was just a MapReduce clone, more or less. And then there was Hadoop 2, which was kind of meant to do everything. And that didn't really work out, so there was specialisation. I don't really see the problem. Many of these do different things, others do similar things different ways. Some are essentially dead ends (I doubt there are many new users of Storm, say). This all seems…

No Hadoop does more than MapReduce, much more. Primarily: MapReduce does not have a distributed file-like block storage.

OK, so, to be more precise:

Hadoop MapReduce does what Google MapReduce does, and Hadoop Distributed File System does what the Google Filesystem does.

And Apache Spark does the same thing as that other Google product whose name I forget.

And Drill vs Dremel, etc etc etc.

I think that the OP is plenty right for the purposes of the point they were making. And I think (from a non-Googler's perspective) that it's a worthwhile point, since the FOSS ecosystem has since evolved along similar lines to Google's internal tools.

Re: Apache Flink

#45

Post by lenticular »

bunderbunder wrote: post #43
lenticular wrote: post #38

Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't need them. I see it all the time with ML work. You wind up using a cluster to overcome the memory inefficiency of Spark, when you could have just used a single machine. For example, I've done huge graph clustering models on a single machine just by being smar…

This has been my experience, too. I worked at just one place that had a really good handle on high-volume, high-velocity streaming data, and they didn't use Flink or Storm or Kafka or anything like that. They mostly just used the KISS principle and a protobuf-style wire format.[1] There is definitely a point where these sorts of scale-out-centric solutions are unavoidable. Short of that point, though, they're probabl…

That's a smart way of doing it. In much ML work, I've found that the hash trick can be used to train many more things in a constant memory, out-of-core fashion than most people think.

Re: Apache Flink

#46

Post by deepsun »

lenticular wrote: post #38

Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't need them. I see it all the time with ML work. You wind up using a cluster to overcome the memory inefficiency of Spark, when you could have just used a single machine. For example, I've done huge graph clustering models on a single machine just by being smar…

I've seen the opposite in our company. We did some data science, on a data that wasn't that big, but big enough to not fit into one machine. Everyone was reluctant to move to Spark, so data-scientists computed their models over subset of data: one day instead of one week of data. And after the project finished, and passed to client, they realized that subset was not representative enough to meet the criteria.

Of course, it's kinda their fault to assume that subset is representative, but if you start checking every assumption you make in data science, then you won't get far.

Had they used Spark from the beginning, they would not have that last-moment surprise, because they would work on proper sized dataset from the beginning.

Re: Apache Flink

#47

Post by riku_iki »

manigandham wrote: post #39
lenticular wrote: post #38

Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't need them. I see it all the time with ML work. You wind up using a cluster to overcome the memory inefficiency of Spark, when you could have just used a single machine. For example, I've done huge graph clustering models on a single machine just by being smar…

We have also seen this. The huge memory capacities now offered in the cloud makes single node processing very capable with entire TB datasets fitting into RAM, and running quickly enough to offset the hourly cost. Spark clusters are more efficient though for longer-running or continuous background processes.

And spark also has some other goodies: integration with storages, monitoring, fault tolerance, common algorithms implemented.

Re: Apache Flink

#48

Post by dvlsg »

continuations wrote: post #15
powerapple wrote: post #12

Earlier quoted context omitted.

... I am using Apache Pulsar to replace current Kafka setup * Apache Pulsar is an open-source distributed pub-sub messaging system

What compelled you to replace Kafka with Pulsar?

Not OP, but I'm keeping a close eye on pulsar.

Im not sure I'd want to replace an existing / working Kafka setup with it, but if pulsar catches on, I would very strongly consider using it for newer projects.

I haven't used pulsar extensively in production for anything, but all the setup and testing I've done on my own was a pleasure. Hoping pulsar continues to grow.

Re: Apache Flink

#49

Post by meatyapp »

What's the easiest way to get started with trying Apache Kafka/Spark/Flink on the cloud? If I want to try out Redis there's RedisLabs, CloudAMQP for RabbitMQ, Compose for Postgres/Redis/RabbitMQ, offerings like Google Cloud SQL/MemoryStore and AWS RDS/ElastiCache, etc. Where do I go for some easy Apache deployments?

Re: Apache Flink

#50

Post by [deleted] »

deepsun wrote: post #46
lenticular wrote: post #38

Controversial opinion here, but all of these distributed streaming architectures are massively overused. They certainly have their place, but you probably don't need them. I see it all the time with ML work. You wind up using a cluster to overcome the memory inefficiency of Spark, when you could have just used a single machine. For example, I've done huge graph clustering models on a single machine just by being smar…

I've seen the opposite in our company. We did some data science, on a data that wasn't that big, but big enough to not fit into one machine. Everyone was reluctant to move to Spark, so data-scientists computed their models over subset of data: one day instead of one week of data. And after the project finished, and passed to client, they realized that subset was not representative enough to meet the criteria. Of cour…

[deleted]
Post reply on HN