Polars: Fast DataFrame library for Rust and Python • Hacker News Board

Re: Polars: Fast DataFrame library for Rust and Python

#61

Post by sriku » Fri, Dec 17, 2021, 3:49 AM UTC

sdfgsdf wrote: post #17

nojito wrote: post #3

Earlier quoted context omitted.

Why? The benchmarks speak volumes. https://h2oai.github.io/db-benchmark/

The benchmarks speak volumes of dishonesty. They sorted the results by speed of 1st run. For a language like Julia, which is JIT-compiled, that's not a fair comparison, considering that you compile once and run millions of times. Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny because once you notice it those benchmarks a…

Agree .. and I was looking for an option to sort by second run.

One trick I've tried to some effect is to run jl code on a smaller data sizes so the compilation gets done and then repeat on the large one so it doesn't get interrupted by compilation. Not sure if this is a recommended approach. Benchmarking Julia is a pain for this reason - compilation always gets mixed up with runtime. But it hasn't prevented me from using it interactively. Pretty happy with it actually.

Re: Polars: Fast DataFrame library for Rust and Python

#62

Post by RcrdBrt » Fri, Dec 17, 2021, 3:55 AM UTC

vatican_banker wrote: post #56

vavooom wrote: post #26

Earlier quoted context omitted.

Also worth plugging the advanced speed of R’s data.table package which continues to trump dplyr to this day. The syntax is also more compact and straightforward once you understand how to query data with it.

In what way data.table trumps dplyr? Genuinely interested in knowing. While data.table is faster than dplyr, data manipulations with data.table are difficult to read/understand/maintain. dplyr also grew into a full-fledge list of libraries to work on data-related projects (the tidyverse). These libraries are _very_ well thought out and enables productivity with minimal learning curve [anecdotal]

Anecdotal data: I found that data.table ingestion speed with fread() trumps absolutely everything else

Re: Polars: Fast DataFrame library for Rust and Python

#63

Post by ttymck » Fri, Dec 17, 2021, 4:19 AM UTC

civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

Is there dplyr API for pandas? That would seem like a very valuable "translation" layer for transitioning or cross language devs. Maybe there is some language barrier to implementing an elegant/faithful version in python?

Re: Polars: Fast DataFrame library for Rust and Python

#64

Post by bogeholm » Fri, Dec 17, 2021, 5:00 AM UTC

riskneutral wrote: post #52

I'm confused. Polars is built on top of the Rust of bindings for Apache Arrow. Arrow already has Python bindings. What does this project add by creating a new Python binding on top of the Rust binding?

Polars is not using Rust bindings for Arrow, it uses a Rust implementation called arrow2: https://github.com/pola-rs/polars/blob/master/polars/polars-...

Arrow2: https://lib.rs/crates/arrow2

Re: Polars: Fast DataFrame library for Rust and Python

#65

Post by fault1 » Fri, Dec 17, 2021, 5:10 AM UTC

pdeffebach wrote: post #22

civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

DataFramesMeta.jl might be exactly what you are looking for then! The syntax is very close to dplyr, but has performance benefits thanks to Julia. Here is a tutorial for those familiar with dplyr: https://juliadata.github.io/DataFramesMeta.jl/stable/dplyr/

DataFramesMeta is great!

But I always get confused by the name. Since DataFrames.jl is lower level shouldn't that be DataFramesBase.jl and the meta package be DataFrames.jl?

Re: Polars: Fast DataFrame library for Rust and Python

#66

Post by 1egg0myegg0 » Fri, Dec 17, 2021, 5:10 AM UTC

rytill wrote: post #35

How would this compare to loading a sqlite database into memory and performing queries with it?

Polars would be 10-100x faster, but so would DuckDB!

Re: Polars: Fast DataFrame library for Rust and Python

#67

Post by 1egg0myegg0 » Fri, Dec 17, 2021, 5:12 AM UTC

adgjlsfhk1 wrote: post #60

nojito wrote: post #36

Earlier quoted context omitted.

>The benchmarks speak volumes of dishonesty. Not really. They are designed to showcase a common use case across multiple technologies. The beauty of this benchmark is that there is a hardware limit included so that it forces you to create novel solutions to perform well. >Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny be…

the benchmarks are a bit out of date (missing DataFrames 1.2/1.3, Julia 1.7, CSV 0.9). I'm planning on running an updated version this weekend.

If you wouldn't mind, please update DuckDB as well!

Re: Polars: Fast DataFrame library for Rust and Python

#68

Post by davnn » Fri, Dec 17, 2021, 5:51 AM UTC

pdeffebach wrote: post #22

civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

DataFramesMeta.jl might be exactly what you are looking for then! The syntax is very close to dplyr, but has performance benefits thanks to Julia. Here is a tutorial for those familiar with dplyr: https://juliadata.github.io/DataFramesMeta.jl/stable/dplyr/

One of the piping macro packages + dataframes.jl works as well.

Re: Polars: Fast DataFrame library for Rust and Python

#69

Post by adgjlsfhk1 » Fri, Dec 17, 2021, 5:57 AM UTC

1egg0myegg0 wrote: post #67

adgjlsfhk1 wrote: post #60

Earlier quoted context omitted.

the benchmarks are a bit out of date (missing DataFrames 1.2/1.3, Julia 1.7, CSV 0.9). I'm planning on running an updated version this weekend.

If you wouldn't mind, please update DuckDB as well!

Can you make a PR to https://github.com/oscardssmith/db-benchmark? I don't know DuckDB, so I don't know what the change would be.

Re: Polars: Fast DataFrame library for Rust and Python

#70

Post by extr » Fri, Dec 17, 2021, 6:02 AM UTC

civilized wrote: post #34

extr wrote: post #15

Earlier quoted context omitted.

dplyr API is not ideal in my experience. Overly verbose and confusing group/melt/cast operators. I much much prefer data.table. In your edit you mention concision, data.table is practically the platonic ideal of that!

Meh. Some people will never stop using Perl or APL because you can get anything done in five random characters (well, anything the language is optimized to express, everything else is a lot harder). I respect it but it's not for me. The tidyverse has the most advanced and intuitive versions of all the things you mention IMO. It has evolved a lot in the past couple years and your impressions of it could be out of date…

I did go check out what's new in the tidyverse after your comment and was pleased to see new functions like pivot_wider and pivot_longer replacing the extremely confusing mess of spread and unite. So it's great to see the ecosystem evolving toward better usability. However I would hardly count it as a victory when late in the game you have to change the API for some core data manipulation functions because you made them too confusing the first time around.

I think you are also maybe assuming everyone has the same use-case as you for data manipulation libraries. If you are coming from a non-programming context and picking up R for the first time, no doubt tidyverse is the way to do that. The verbosity is obviously a benefit if you're having to read someone else's code and are not interested in learning a DSL just to understand what columns are being filtered on or dropped or whatever.

But if you are doing data analysis full time and are writing thousands of lines of throwaway EDA code a week, most of it only to be seen by yourself, the concision and speed that data.table offers is basically second to none, in any language. Rapid iteration for you personally is the point. Less typing is good, because you're trying to move as fast as possible to explore hypotheses. Execution speed on medium sized data is important, because a few extra seconds on every run matters a lot when you are running 500 micro-batches of analysis code a day. And as the h2o benchmarks show, data.table is still quite a bit faster than dplyr. Obviously not everyone needs the speed, but a lot of us do!