Imagine if data bricks switched and just started to contribute to this.
I live in a dream world :)
DataFusion Comet: Apache Spark Accelerator
11–20 of 26 posts
Re: DataFusion Comet: Apache Spark Accelerator
#12Earlier quoted context omitted.
The OP is the original creator of Ballista, so he's well aware of the project. Ballista is much less mature than Spark and needs a lot of work. It's awesome they're making Spark faster with Comet.
Yes, Ballista failed to gain traction. I think that one of the challenges was that it only supported a small subset of Spark, and there was too much work involved to try and get to parity with Spark. The Comet approach is much more pragmatic because we just add support for more operators and expressions over time and fall back to Spark for anything that is not supported yet.
For the longest time, searching for Ballista linked to its old archived repo that didn't even have a link to the new repo. There was no search result for the new repo. This misled people into thinking that Ballista is a dead project but it wasn't. It wasted so much opportunity.
I don't think it's a fair criticism of Ballista to say that it failed in any way. It just looks to need substantial effort to bring it on par with Spark. The performance benefits are meaningful. Ballista can then not only take the crown from Spark, but also revalidate Rust as a language.
Re: DataFusion Comet: Apache Spark Accelerator
#13Imagine if data bricks switched and just started to contribute to this. I live in a dream world :)
They have their own implementation that is closed source (last time I checked), Photon [1], which is written with a C++ engine.
Databricks' terms prevent(ed?) publishing benchmarks, it would be interesting to see how Comet performs relative to it over time.
Photon comes at a higher cost, so one big advantage of Comet is being able to deploy it on a standard Databricks cluster that doesn't have Photon, at a lower running cost.
Re: DataFusion Comet: Apache Spark Accelerator
#14Earlier quoted context omitted.
Yes, Ballista failed to gain traction. I think that one of the challenges was that it only supported a small subset of Spark, and there was too much work involved to try and get to parity with Spark. The Comet approach is much more pragmatic because we just add support for more operators and expressions over time and fall back to Spark for anything that is not supported yet.
For the longest time, searching for Ballista linked to its old archived repo that didn't even have a link to the new repo. There was no search result for the new repo. This misled people into thinking that Ballista is a dead project but it wasn't. It wasted so much opportunity. I don't think it's a fair criticism of Ballista to say that it failed in any way. It just looks to need substantial effort to bring it on par…
I wish I'd known about the search issue.
I do see a new opportunity for Ballista. By leveraging all of the Spark-compatible operators and expressions being built in Comet, it would be able to support a wider range of queries much more quickly.
Ballista already uses protobuf for sending plans to executors and Comet accepts protobuf plans (in a similar, but different format).
Re: DataFusion Comet: Apache Spark Accelerator
#15Earlier quoted context omitted.
Lots of Spark workloads are executed with the C++ Photon engine on the Databricks platform, so we ironically have partially moved back to C++. Disclosure: I work for Databricks.
The continued use of C++ is not exactly something to be proud of, although in this case at least it presumably is for short-running jobs, not for long-running services that accumulate leaks.
There is a ton of reliable load-bearing software out there written in C++. I don't think the fact that a piece of software is written in C++ is enough to presume that it has memory leaks.
Re: DataFusion Comet: Apache Spark Accelerator
#16But why. Unless you need to use low-level map/reduce, just ditch Spark and use https://github.com/apache/datafusion-ballista directly. It supports Python too.
There is an entire ecosystem of libraries for Spark built up over years.
I want to be able to connect to interact with the full services from GCS, Azure, AWS, OpenAI etc none of which DataFusion supports.
As well as use libraries such as SynapseML, SparkNLP etc.
And do all of this with full support from my cloud provider.
Re: DataFusion Comet: Apache Spark Accelerator
#17Earlier quoted context omitted.
The OP is the original creator of Ballista, so he's well aware of the project. Ballista is much less mature than Spark and needs a lot of work. It's awesome they're making Spark faster with Comet.
Yes, Ballista failed to gain traction. I think that one of the challenges was that it only supported a small subset of Spark, and there was too much work involved to try and get to parity with Spark. The Comet approach is much more pragmatic because we just add support for more operators and expressions over time and fall back to Spark for anything that is not supported yet.
One of the challenges is that most Spark users don't care if you 2x performance.
We are in the enterprise with large cloud budgets and can simply change instance types. If you're 20x then that is a different story but then (a) you need to have feature parity and (b) need support from cloud vendors which Spark has.
Re: DataFusion Comet: Apache Spark Accelerator
#18Earlier quoted context omitted.
Lots of Spark workloads are executed with the C++ Photon engine on the Databricks platform, so we ironically have partially moved back to C++. Disclosure: I work for Databricks.
The continued use of C++ is not exactly something to be proud of, although in this case at least it presumably is for short-running jobs, not for long-running services that accumulate leaks.
Python would be just another PHP level language if it wasn't for C++.
It's what powers all of the DE/ML/AI libraries.
Re: DataFusion Comet: Apache Spark Accelerator
#19Imagine if data bricks switched and just started to contribute to this. I live in a dream world :)
They have their own implementation that is closed source (last time I checked), Photon [1], which is written with a C++ engine. Databricks' terms prevent(ed?) publishing benchmarks, it would be interesting to see how Comet performs relative to it over time. Photon comes at a higher cost, so one big advantage of Comet is being able to deploy it on a standard Databricks cluster that doesn't have Photon, at a lower runn…
nice word play for two competing spark execution engines: Photon and Comet.
C++ vs Rust, closed-source vs open-source
Re: DataFusion Comet: Apache Spark Accelerator
#20Earlier quoted context omitted.
For the longest time, searching for Ballista linked to its old archived repo that didn't even have a link to the new repo. There was no search result for the new repo. This misled people into thinking that Ballista is a dead project but it wasn't. It wasted so much opportunity. I don't think it's a fair criticism of Ballista to say that it failed in any way. It just looks to need substantial effort to bring it on par…
I wish I'd known about the search issue. I do see a new opportunity for Ballista. By leveraging all of the Spark-compatible operators and expressions being built in Comet, it would be able to support a wider range of queries much more quickly. Ballista already uses protobuf for sending plans to executors and Comet accepts protobuf plans (in a similar, but different format).
Did Databricks sponsor Comet?