Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Daft: A High-Performance Distributed Dataframe Library for Multimodal Data

blog.getdaft.io

Post reply on HN
Print view View original

31–38 of 38 posts

Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data

#31

Post by esafak »

marsupialtail_2 wrote: post #24
Fiahil wrote: post #10

So Daft is a distributed Polars ? If we set apart the distributed part, what's the "killer feature" of Daft for trying to compete with Polars (and Pandas) ? Are they API-compatible ? How's the memory consumption benchmark ? (TBH, this is the only interesting metric. Timing and Latency are not really important when your most important competitor is Spark)

Hi -- I am the author of Quokka: https://github.com/marsupialtail/quokka , trying to be distributed Polars. I am trying to go for API compatibility, or at least supporting most of the API. I am not focused on complex data types though.

Same suggestion to you: please integrate it with fugue.

https://github.com/fugue-project/fugue

Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data

#32

Post by [deleted] »

Fiahil wrote: post #20
jaychia wrote: post #16

Earlier quoted context omitted.

> So Daft is a distributed Polars ? We did actually start by using Polars as our underlying execution engine, but eventually transitioned off to our own Rust Table abstraction to better suit our needs (e.g. custom datatypes and kernels). We still share the arrow2 dependency with Polars for in-memory representation of our data. > what's the "killer feature" of Daft for trying to compete with Polars (and Pandas) We don…

Thanks you for your responses ! I'm going to be very blunt here, because you need to hear this to go forward : You HAVE TO be at least API-compatible with Polars or Pandas to exist. Being backend-compatible with arrow is not enough. There is no technical reason why you would not pick one and go with it, apart from being a very difficult task. As of today, I have 2 major pains : Pandas being a giant memory hog and Pol…

[deleted]

Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data

#33

Post by liminal »

jaychia wrote: post #19
liminal wrote: post #15

This looks amazing. * I started poking around the docs and I'm most excited about a Ray backend runner. I'm hoping this allows more ergonomic distributed data frame computation on an existing Ray cluster. * Is this based on Apache Arrow? I would assume so, but it's important that it be zero-copy from other tools. Would like to see this mentioned prominently somewhere. * I really like the Polars expression API. I have…

> Ray backend runner Yes, give it a whirl and let us know what you think! Ray is amazing and has actually gotten a lot better post their 2.0 release :) > Is this based on Apache Arrow? Indeed it is, and thanks for the feedback. We'll make this a little more visible. We use the arrow2 Rust crate (same one that Polars uses) for our in-memory data representation. Our data representation makes it such that converting Daf…

Please at least reach out to the Polars folks and see what's possible.

Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data

#34

Post by nemoniac »

jaychia wrote: post #18
silveraxe93 wrote: post #13

From past experience, the hackernews crowd is super hostile to tools/etc with opt-out [telemetry]( https://www.getdaft.io/projects/docs/en/latest/telemetry.htm... ).

We hear you, and thanks for making this visible! As a performance-driven project it’s important for us to understand which operations and use-cases are slowest/buggiest for our users so that we can focus on them. We tried to be very intentional in scoping the telemetry we collect and take this very seriously (telemetry is top-level on both our docs and README). Happy to hear any feedback on this - we understand it's…

Since you ask, some feedback: opt-out on telemetry is in contravention of EU law. It applies when any EU citizen uses your software.

"...consent options structured as an opt-out selected by default is a violation of the GDPR..."

https://en.wikipedia.org/wiki/General_Data_Protection_Regula...

Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data

#35

Post by IanOzsvald »

jaychia wrote: post #9
swader999 wrote: post #6

Finally a decent name.

Hello! I am one of the maintainers of Daft. Funny enough I just gave a presentation about Daft in London and we all had quite a laugh at the name :D

Indeed I was the one who got confused by the name! Thanks for attending the discussion Jay and I'm happy to see Daft being discussed here

Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data

#36

Post by zekrioca »

jaychia wrote: post #18
silveraxe93 wrote: post #13

From past experience, the hackernews crowd is super hostile to tools/etc with opt-out [telemetry]( https://www.getdaft.io/projects/docs/en/latest/telemetry.htm... ).

We hear you, and thanks for making this visible! As a performance-driven project it’s important for us to understand which operations and use-cases are slowest/buggiest for our users so that we can focus on them. We tried to be very intentional in scoping the telemetry we collect and take this very seriously (telemetry is top-level on both our docs and README). Happy to hear any feedback on this - we understand it's…

Instead of having your own opt-out, why don't you simply ask your users? Do a simple survey after some time a user is using your product. Offer them a cup of coffee, and they will gladly send you some real, genuine feedback.

Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data

#37

Post by camgunz »

jaychia wrote: post #11
camgunz wrote: post #2

1. This looks super cool 2. I admit to not understanding data lakes at all. I thought it was like a failure case for like, "we can't figure out how to get this data into a database", because isn't updating it a huge chore? You have to make sure that if you're updating you're not also generating new analytics, which it seems like you're always doing because it's very slow. Don't databases solve this pretty elegantly?…

Hi, I'm one of the maintainers of Daft 1. Thanks! We think so too :) 2. Here's my 2c in argument of flat files - Ingestion: ingesting things into a data lake is much easier than writing to a database (all you have to do is drop some JSON, CSVs or protobufs into a bucket). This makes integrating with other systems, especially 3rd-party or vendors, much easier since there's an open language-agnostic format to communica…

Heya thanks for the super informative response! I learned a lot here it was very useful. Good luck with Daft!

Re: Daft: A High-Performance Distributed Dataframe Library for Multimodal Data

#38

Post by Fiahil »

sammysidhu wrote: post #29
Fiahil wrote: post #20

Earlier quoted context omitted.

Thanks you for your responses ! I'm going to be very blunt here, because you need to hear this to go forward : You HAVE TO be at least API-compatible with Polars or Pandas to exist. Being backend-compatible with arrow is not enough. There is no technical reason why you would not pick one and go with it, apart from being a very difficult task. As of today, I have 2 major pains : Pandas being a giant memory hog and Pol…

Hi! (one of the Daft maintainers here), thanks for the feedback. Ultimately you're right that supporting the full Polars syntax in a distributed fashion is very difficult. There are libraries out there that do "Pandas but distributed" but from what I have seen is that they prioritized API coverage rather than performance or memory consumption. So you end up in a similar boat to the situation you mentioned. We're tryi…

Then, maybe the right choice isn't to start a fresh DataFrame library from arrow, but rather leverage Polars and build out the distributed part (in Rust, of course, not in Python).

> We're trying to start with a simpler API that maps well to a distributed query query that we can execute well and then add the features that people request for.

That would have been a good approach on a field that has not been standardised around a single library since its infancy. Polars is beating Pandas in every possible benchmark, yet will continue to struggle for adoption "until the end". Do you really think Daft can do better ? (If yes, go ahaid, and prove me wrong !)

As a comparison, it's like trying to introduce a new transport layer protocol (https://en.wikipedia.org/wiki/QUIC) against TCP. You can do that if and only if there are obvious benefits, no drawbacks and you are prepared to wait 15 years for 30% market share.

Post reply on HN