Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polars: Fast DataFrame library for Rust and Python

pola.rs

Post reply on HN
Print view View original

21–30 of 131 posts

Re: Polars: Fast DataFrame library for Rust and Python

#21

Post by nas »

lern_too_spel wrote: post #10
nas wrote: post #2

It looks interesting but phrases like "embarrassingly parallel execution" make my marketing hype detectors trigger. Maybe they could tone down their self promotion just a touch. Also "Even though Polars is completely written in Rust (no runtime overhead!) ...". I find that hard to believe.

"Embarrassingly parallel" is a technical term, not a marketing term. https://en.wikipedia.org/wiki/Embarrassingly_parallel

It's a term for the nature of a problem, not a library or software package. It looks like they have designed the API so that "embarrassingly parallel" problems can naturally be computed using Polars. That would be fantastic, much better than Pandas. The way they write it sounds like marketing fluff to me and that's a shame because Polars looks like a useful thing.

Re: Polars: Fast DataFrame library for Rust and Python

#22

Post by pdeffebach »

civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

DataFramesMeta.jl might be exactly what you are looking for then! The syntax is very close to dplyr, but has performance benefits thanks to Julia.

Here is a tutorial for those familiar with dplyr: https://juliadata.github.io/DataFramesMeta.jl/stable/dplyr/

Re: Polars: Fast DataFrame library for Rust and Python

#24

Post by cabalamat »

civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

> dplyr

Ths s lbrry whs nm nds mr vwls. F m tlkng t smn, hw m sppsd t prnc t?

Re: Polars: Fast DataFrame library for Rust and Python

#25

Post by goodside »

nas wrote: post #21
lern_too_spel wrote: post #10

Earlier quoted context omitted.

"Embarrassingly parallel" is a technical term, not a marketing term. https://en.wikipedia.org/wiki/Embarrassingly_parallel

It's a term for the nature of a problem, not a library or software package. It looks like they have designed the API so that "embarrassingly parallel" problems can naturally be computed using Polars. That would be fantastic, much better than Pandas. The way they write it sounds like marketing fluff to me and that's a shame because Polars looks like a useful thing.

“Embarrassingly parallel execution” means that it parallelizes (only) problems that are embarrassingly parallel. The meaning is clear — if you want to be really pedantic about it, problems are “parallelizable” and only execution is “parallel”, but “embarrassingly parallelizable” is too many syllables.

Re: Polars: Fast DataFrame library for Rust and Python

#26

Post by vavooom »

civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

Also worth plugging the advanced speed of R’s data.table package which continues to trump dplyr to this day. The syntax is also more compact and straightforward once you understand how to query data with it.

Re: Polars: Fast DataFrame library for Rust and Python

#28

Post by eternalban »

ddavis wrote: post #8
abeppu wrote: post #7

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.

There is an effort for this: https://github.com/data-apis/dataframe-api

https://data-apis.org/dataframe-protocol/latest/design_requi...

Re: Polars: Fast DataFrame library for Rust and Python

#29

Post by paulgb »

sdfgsdf wrote: post #17
nojito wrote: post #3

Earlier quoted context omitted.

Why? The benchmarks speak volumes. https://h2oai.github.io/db-benchmark/

The benchmarks speak volumes of dishonesty. They sorted the results by speed of 1st run. For a language like Julia, which is JIT-compiled, that's not a fair comparison, considering that you compile once and run millions of times. Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny because once you notice it those benchmarks a…

> considering that you compile once and run millions of times.

If you’re writing data pipelines then yes, but a lot of Pandas users use it interactivity. As much as I’d rather use Julia, the last time I tried it I found myself waiting for computation far more often than with a Jupyter/Python workflow.

Re: Polars: Fast DataFrame library for Rust and Python

#30

Post by devin-petersohn »

abeppu wrote: post #7

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.

This was my PhD focus. We identified a core "dataframe algebra"[1] that encompasses all of pandas (and R/S data.frames): a total of 16 operators that cover all 600+ operators of pandas. What you describe was exactly our aim. It turns out there are a lot of operators that are really easy to support and make fast, and that gets you about 60% or so of the way to supporting all of pandas. Then there are really complex operators that may alter the schema in a way that is undeterminable before the operation is carried out (think a row-wise or column-wise `df.apply`). The flexibility that pandas offers is something we were able to express mathematically, and with that math we can start to optimize the dataframe holistically, rather than chipping away at small parts of pandas that are embarrassingly parallel.

Most dataframe libraries cannot architecturally support the entire dataframe algebra and data model because they are optimized for specific use-cases (which is not a bad thing). It can be frustrating for users who may have no idea what they can do with a given tool just because it is called "dataframe", but I don't know how to fix that.

[1] https://arxiv.org/pdf/2001.00888

Post reply on HN