Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DataFusion Comet: Apache Spark Accelerator

github.com

Post reply on HN
Print view View original

21–26 of 26 posts

Re: DataFusion Comet: Apache Spark Accelerator

#21

Post by spenczar5 »

andygrove wrote: post #10
MrPowers wrote: post #5

Earlier quoted context omitted.

The OP is the original creator of Ballista, so he's well aware of the project. Ballista is much less mature than Spark and needs a lot of work. It's awesome they're making Spark faster with Comet.

Yes, Ballista failed to gain traction. I think that one of the challenges was that it only supported a small subset of Spark, and there was too much work involved to try and get to parity with Spark. The Comet approach is much more pragmatic because we just add support for more operators and expressions over time and fall back to Spark for anything that is not supported yet.

There seems to be a history of data technologies requiring a serious corporate sponsor. Arrow gets so much dev and marketing effort from Voltron, Spark from Databricks, etc. Did Ballista have anything’s similar? I loved the project but it never seemed to move very fast on integrating with other tools and platforms.

Re: DataFusion Comet: Apache Spark Accelerator

#22

Post by pitah1 »

I've been keeping an eye on these kinds of Spark accelerator libraries for a while now.

How does it compare to Blaze[1] and Gluten[2]?

I'm interested in running some benchmarks soon against all three for my project to see how they all go.

[1] https://github.com/kwai/blaze

[2] https://github.com/apache/incubator-gluten

Re: DataFusion Comet: Apache Spark Accelerator

#23

Post by andygrove »

OutOfHere wrote: post #20
andygrove wrote: post #14

Earlier quoted context omitted.

I wish I'd known about the search issue. I do see a new opportunity for Ballista. By leveraging all of the Spark-compatible operators and expressions being built in Comet, it would be able to support a wider range of queries much more quickly. Ballista already uses protobuf for sending plans to executors and Comet accepts protobuf plans (in a similar, but different format).

Did Databricks sponsor Comet?

No.

Re: DataFusion Comet: Apache Spark Accelerator

#24

Post by nicornk »

orthoxerox wrote: post #8
OutOfHere wrote: post #2

But why. Unless you need to use low-level map/reduce, just ditch Spark and use https://github.com/apache/datafusion-ballista directly. It supports Python too.

Because it's a drop-in replacement that lets you (theoretically) spend O(1) development effort on speeding up your Spark jobs instead of O(N). I say theoretically, because I have no idea how Comet works with the memory limits on Spark executors. If you have to rebalance the memory between regular memory and memory overhead or provision some off-heap memory for Comet, then the migration won't be so simple.

and this is unfortunately the case for Velox.

Re: DataFusion Comet: Apache Spark Accelerator

#25

Post by ed_elliott_asc »

pitah1 wrote: post #22

I've been keeping an eye on these kinds of Spark accelerator libraries for a while now. How does it compare to Blaze[1] and Gluten[2]? I'm interested in running some benchmarks soon against all three for my project to see how they all go. [1] https://github.com/kwai/blaze [2] https://github.com/apache/incubator-gluten

Apparently blaze is also datafusion

Re: DataFusion Comet: Apache Spark Accelerator

#26

Post by _zoltan_ »

nicornk wrote: post #24
orthoxerox wrote: post #8

Earlier quoted context omitted.

Because it's a drop-in replacement that lets you (theoretically) spend O(1) development effort on speeding up your Spark jobs instead of O(N). I say theoretically, because I have no idea how Comet works with the memory limits on Spark executors. If you have to rebalance the memory between regular memory and memory overhead or provision some off-heap memory for Comet, then the migration won't be so simple.

and this is unfortunately the case for Velox.

care to elaborate?
Post reply on HN