Viewing profile — andygrove

I've just started exploring adding OpenTelemetry support to the Comet subproject of DataFusion. I'm excited to see the integration with Apache Arrow (Rust) and potentially DataFusi…

Congrats on the launch! I contributed to the NVIDIA Spark RAPIDS project for ~4 years and for the past year have been contributing to DataFusion Comet, so I have some experience in…

No.

I wish I'd known about the search issue. I do see a new opportunity for Ballista. By leveraging all of the Spark-compatible operators and expressions being built in Comet, it would…

Yes, Ballista failed to gain traction. I think that one of the challenges was that it only supported a small subset of Spark, and there was too much work involved to try and get to…

Original author of DataFusion/Ballista here. Having alamb and others from InfluxData involved has been a huge help in driving the project forward and helping build an active commun…

I will update these old pages on my blog and redirect them!

I think it is worth pointing out that this tool does support querying Delta Lake (the author of ROAPI is also a major contributor the native Rust implementation of Delta Lake). Del…

FWIW, my dealer is not adding any markup. Here is the email I received from them explaining the process: "We finally received an update from Ford regarding the Lightning! This will…

The Apache Spark project is many many years ahead of DataFusion & Ballista with more than a decade of work from more than 1,700 contributors and is going strong. I don't see DataFu…

There is no support for Plasma. There is a TableProvider API for custom file formats and there is built-in support for CSV, Parquet, and JSON.

Ballista started out as a separate project and was donated in April 2021. They currently share a release schedule (but have different versioning) and this was the first release of …

The best "Getting Started" documentation right now is that on docs.rs - https://docs.rs/ballista/0.5.0/ballista/ This demonstrates using the Rust client (BallistaContext + DataFram…

Yes. The Ballista crate (part of the arrow-datafusion repo) provides distributed query execution and the scheduler has a gRPC service. Flight is used internally as well but not dir…

There is experimental support for distributed query execution with spill-to-disk between stages to support larger than memory datasets. This is implemented in the Ballista crate, w…

That doesn't surprise me at all. I have politely requested that the project considers renaming to avoid this. https://github.com/datafuselabs/datafuse/issues/654

I thought this was about me until I got to the word Rich in the title.

Ah, yes, that makes sense. I can see how this could have been misread.

There is also a more recent blog post which perhaps led to the project being posted here (I am guessing). https://andygrove.io/2021/01/ballista-2021/

The project uses stable Rust. Which version are you trying to compile with?