DataFusion Comet: Apache Spark Accelerator
1–10 of 26 posts
Re: DataFusion Comet: Apache Spark Accelerator
#2But why. Unless you need to use low-level map/reduce, just ditch Spark and use https://github.com/apache/datafusion-ballista directly. It supports Python too.
Re: DataFusion Comet: Apache Spark Accelerator
#3But why. Unless you need to use low-level map/reduce, just ditch Spark and use https://github.com/apache/datafusion-ballista directly. It supports Python too.
Many companies have 100k+ of lines of Spark code. It's not trivial to rewrite all of that in another query framework.
Re: DataFusion Comet: Apache Spark Accelerator
#4But why. Unless you need to use low-level map/reduce, just ditch Spark and use https://github.com/apache/datafusion-ballista directly. It supports Python too.
Many companies have 100k+ of lines of Spark code. It's not trivial to rewrite all of that in another query framework.
Following that logic, we should have stuck with C/C++ for everything. /s
Re: DataFusion Comet: Apache Spark Accelerator
#5But why. Unless you need to use low-level map/reduce, just ditch Spark and use https://github.com/apache/datafusion-ballista directly. It supports Python too.
The OP is the original creator of Ballista, so he's well aware of the project.
Ballista is much less mature than Spark and needs a lot of work. It's awesome they're making Spark faster with Comet.
Re: DataFusion Comet: Apache Spark Accelerator
#6Earlier quoted context omitted.
Many companies have 100k+ of lines of Spark code. It's not trivial to rewrite all of that in another query framework.
Following that logic, we should have stuck with C/C++ for everything. /s
Lots of Spark workloads are executed with the C++ Photon engine on the Databricks platform, so we ironically have partially moved back to C++. Disclosure: I work for Databricks.
Re: DataFusion Comet: Apache Spark Accelerator
#7Earlier quoted context omitted.
Following that logic, we should have stuck with C/C++ for everything. /s
Lots of Spark workloads are executed with the C++ Photon engine on the Databricks platform, so we ironically have partially moved back to C++. Disclosure: I work for Databricks.
The continued use of C++ is not exactly something to be proud of, although in this case at least it presumably is for short-running jobs, not for long-running services that accumulate leaks.
Re: DataFusion Comet: Apache Spark Accelerator
#8But why. Unless you need to use low-level map/reduce, just ditch Spark and use https://github.com/apache/datafusion-ballista directly. It supports Python too.
Because it's a drop-in replacement that lets you (theoretically) spend O(1) development effort on speeding up your Spark jobs instead of O(N).
I say theoretically, because I have no idea how Comet works with the memory limits on Spark executors. If you have to rebalance the memory between regular memory and memory overhead or provision some off-heap memory for Comet, then the migration won't be so simple.
Re: DataFusion Comet: Apache Spark Accelerator
#9But why. Unless you need to use low-level map/reduce, just ditch Spark and use https://github.com/apache/datafusion-ballista directly. It supports Python too.
In short: Compatible with existing Spark jobs but executing them much faster. Benchmarks in the README file and docs [1] show improvements up to 3x while not even all operations are implemented yet (i.e. if an operation is not available in Comet it falls back to Spark), so there is room for further improvements. Across all TPC-H queries the total speedup is currently 1.5x, the docs state that based on datafusion's standalone performance 2x-4x is a realistic goal [1]
Haven't seen any memory consumption benchmarks but suspect that it's lower than Spark for same jobs since datafusion is designsd from the ground up to be columnar-first.
For companies spending 100s of thousands if not millions on compute this would mean substantial savings with little effort.
[1] https://datafusion.apache.org/comet/contributor-guide/benchm...
Re: DataFusion Comet: Apache Spark Accelerator
#10But why. Unless you need to use low-level map/reduce, just ditch Spark and use https://github.com/apache/datafusion-ballista directly. It supports Python too.
The OP is the original creator of Ballista, so he's well aware of the project. Ballista is much less mature than Spark and needs a lot of work. It's awesome they're making Spark faster with Comet.
Yes, Ballista failed to gain traction. I think that one of the challenges was that it only supported a small subset of Spark, and there was too much work involved to try and get to parity with Spark.
The Comet approach is much more pragmatic because we just add support for more operators and expressions over time and fall back to Spark for anything that is not supported yet.