Finally a decent name.
Hello! I am one of the maintainers of Daft. Funny enough I just gave a presentation about Daft in London and we all had quite a laugh at the name :D
one of the best bands in the world has Daft in their name!
Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
21–30 of 38 posts
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#22From past experience, the hackernews crowd is super hostile to tools/etc with opt-out [telemetry]( https://www.getdaft.io/projects/docs/en/latest/telemetry.htm... ).
We hear you, and thanks for making this visible! As a performance-driven project it’s important for us to understand which operations and use-cases are slowest/buggiest for our users so that we can focus on them. We tried to be very intentional in scoping the telemetry we collect and take this very seriously (telemetry is top-level on both our docs and README). Happy to hear any feedback on this - we understand it's…
tbh it _was_ quite easy to spot.
The gold standard would be making it opt-in, but I can guess barely no-one would enable it.
I don't have strong opinions either way in this. But it's usually something that sparks a big flaming thread.
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#23From past experience, the hackernews crowd is super hostile to tools/etc with opt-out [telemetry]( https://www.getdaft.io/projects/docs/en/latest/telemetry.htm... ).
We hear you, and thanks for making this visible! As a performance-driven project it’s important for us to understand which operations and use-cases are slowest/buggiest for our users so that we can focus on them. We tried to be very intentional in scoping the telemetry we collect and take this very seriously (telemetry is top-level on both our docs and README). Happy to hear any feedback on this - we understand it's…
Instead of having your own vanity opt-out, please do consider https://consoledonottrack.com/ which allows the user to express their intentions without their bashrc growing with every new project
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#24So Daft is a distributed Polars ? If we set apart the distributed part, what's the "killer feature" of Daft for trying to compete with Polars (and Pandas) ? Are they API-compatible ? How's the memory consumption benchmark ? (TBH, this is the only interesting metric. Timing and Latency are not really important when your most important competitor is Spark)
Hi -- I am the author of Quokka: https://github.com/marsupialtail/quokka, trying to be distributed Polars. I am trying to go for API compatibility, or at least supporting most of the API.
I am not focused on complex data types though.
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#251. This looks super cool 2. I admit to not understanding data lakes at all. I thought it was like a failure case for like, "we can't figure out how to get this data into a database", because isn't updating it a huge chore? You have to make sure that if you're updating you're not also generating new analytics, which it seems like you're always doing because it's very slow. Don't databases solve this pretty elegantly?…
While we are on this topic, the challenge with data lakes for Python based projects like Daft and Quokka (what I work on) is the poor Python support for data lakes like Delta, Iceberg and Hudi. Delta has the best support but its Python API is consistently behind the Java ones. Iceberg doesn't support Python writes. Hudi doesn't support anything Python.
I have users demanding Iceberg writes and Hudi reads/writes. I don't know what to tell them, since I don't have the resources to add a reader/writer myself for those projects.
Hopefully as DuckDB becomes more popular we will see Python bindings for these popular data lake formats this year.
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#26Does Daft support Delta table format? Any plans to support sql queries?
SQL support is very challenging.
I work on Quokka (https://github.com/marsupialtail/quokka). I support Iceberg reads. Recently we are adding SQL support from just parsing the DuckDB logical plan, though that is very challenging as well.
The Python world lacks a standard for a plug and play SQL query optimizer. Apache Calcite is good for the JVM world, but not great if you are trying to cut out the JVM.
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#27Earlier quoted context omitted.
We hear you, and thanks for making this visible! As a performance-driven project it’s important for us to understand which operations and use-cases are slowest/buggiest for our users so that we can focus on them. We tried to be very intentional in scoping the telemetry we collect and take this very seriously (telemetry is top-level on both our docs and README). Happy to hear any feedback on this - we understand it's…
Instead of having your own vanity opt-out, please do consider https://consoledonottrack.com/ which allows the user to express their intentions without their bashrc growing with every new project
Hi (one of the maintainers here), that is a good suggestion! I wasn't aware of that project. I went ahead and made an issue to add `export DO_NOT_TRACK=1` as one of the variables we track! https://github.com/Eventual-Inc/Daft/issues/1015
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#28Please integrate it with Fugue; a unified interface that lets you swap out execution engines like pandas for Spark.
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#29Earlier quoted context omitted.
> So Daft is a distributed Polars ? We did actually start by using Polars as our underlying execution engine, but eventually transitioned off to our own Rust Table abstraction to better suit our needs (e.g. custom datatypes and kernels). We still share the arrow2 dependency with Polars for in-memory representation of our data. > what's the "killer feature" of Daft for trying to compete with Polars (and Pandas) We don…
Thanks you for your responses ! I'm going to be very blunt here, because you need to hear this to go forward : You HAVE TO be at least API-compatible with Polars or Pandas to exist. Being backend-compatible with arrow is not enough. There is no technical reason why you would not pick one and go with it, apart from being a very difficult task. As of today, I have 2 major pains : Pandas being a giant memory hog and Pol…
Hi! (one of the Daft maintainers here), thanks for the feedback. Ultimately you're right that supporting the full Polars syntax in a distributed fashion is very difficult. There are libraries out there that do "Pandas but distributed" but from what I have seen is that they prioritized API coverage rather than performance or memory consumption. So you end up in a similar boat to the situation you mentioned.
We're trying to start with a simpler API that maps well to a distributed query query that we can execute well and then add the features that people request for.
I would love to know what you would want to see in Daft!
Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data
#30Please integrate it with Fugue; a unified interface that lets you swap out execution engines like pandas for Spark. https://github.com/fugue-project/fugue
(one of the Daft maintainers here) Great call out! I went ahead and make an issue for us to work on this: https://github.com/Eventual-Inc/Daft/issues/1016