Polars: Fast DataFrame library for Rust and Python

Post by matham » Fri, Dec 17, 2021, 1:39 PM UTC

otsaloma wrote: post #104

civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

Agreed, dplyr is great. I built my own data frame implementation on top of NumPy specifically trying to accomplish a better API, similar to dplyr. It's not exactly the same naming or operations, but should feel familiar and much simpler and consistent than Pandas. And no indexes or axes. Having done this, a couple notes on what will unavoidably differ in Python * It probably makes more sense in Python to use classes,…

Post by nojito » Fri, Dec 17, 2021, 3:14 PM UTC

vatican_banker wrote: post #56

vavooom wrote: post #26

Earlier quoted context omitted.

Also worth plugging the advanced speed of R’s data.table package which continues to trump dplyr to this day. The syntax is also more compact and straightforward once you understand how to query data with it.

In what way data.table trumps dplyr? Genuinely interested in knowing. While data.table is faster than dplyr, data manipulations with data.table are difficult to read/understand/maintain. dplyr also grew into a full-fledge list of libraries to work on data-related projects (the tidyverse). These libraries are _very_ well thought out and enables productivity with minimal learning curve [anecdotal]

    Dt[rows, columns, groups]

Post by chrisaycock » Fri, Dec 17, 2021, 4:06 PM UTC

abeppu wrote: post #7

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.

Post by chrisaycock » Fri, Dec 17, 2021, 4:14 PM UTC

pvitz wrote: post #100

alexisread wrote: post #84

Earlier quoted context omitted.

I believe Vaex can do this, in addition to GPU processing and reading direct from s3. https://github.com/vaexio/vaex

To you and all the other sibling comments: Thanks a lot! Exactly what I have been looking for! With regard to Vaex, I would really be interested in an independent benchmark comparing it to dask, spark, data.table etc. However, I have seen in the comments that others also can't find that.

Post by Hasnep » Fri, Dec 17, 2021, 7:15 PM UTC

fault1 wrote: post #65

pdeffebach wrote: post #22

Earlier quoted context omitted.

DataFramesMeta.jl might be exactly what you are looking for then! The syntax is very close to dplyr, but has performance benefits thanks to Julia. Here is a tutorial for those familiar with dplyr: https://juliadata.github.io/DataFramesMeta.jl/stable/dplyr/

DataFramesMeta is great! But I always get confused by the name. Since DataFrames.jl is lower level shouldn't that be DataFramesBase.jl and the meta package be DataFrames.jl?