Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polars: Fast DataFrame library for Rust and Python

pola.rs

Post reply on HN
Print view View original

41–50 of 131 posts

Re: Polars: Fast DataFrame library for Rust and Python

#41

Post by temp8964 »

vavooom wrote: post #26
civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

Also worth plugging the advanced speed of R’s data.table package which continues to trump dplyr to this day. The syntax is also more compact and straightforward once you understand how to query data with it.

The easiest to understand data frame API syntax is SQL: select cols from df where rows match condition group by grouping cols.

data.table syntax is just like that. But less verbose. Plus super fast. No reason to not love it.

Re: Polars: Fast DataFrame library for Rust and Python

#42

Post by gpderetta »

abeppu wrote: post #7

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.

I believe Codd took a stab at it a few years ago. He had some success, but didn't break in data science.

Re: Polars: Fast DataFrame library for Rust and Python

#43

Post by civilized »

temp8964 wrote: post #41
vavooom wrote: post #26

Earlier quoted context omitted.

Also worth plugging the advanced speed of R’s data.table package which continues to trump dplyr to this day. The syntax is also more compact and straightforward once you understand how to query data with it.

The easiest to understand data frame API syntax is SQL: select cols from df where rows match condition group by grouping cols. data.table syntax is just like that. But less verbose. Plus super fast. No reason to not love it.

I agree that if that's all you do with data, data.table makes it easy.

Re: Polars: Fast DataFrame library for Rust and Python

#44

Post by temp8964 »

[deleted] wrote: post #39
nojito wrote: post #37

Earlier quoted context omitted.

The FAQ isn't for new data.table users. https://rdatatable.gitlab.io/data.table/articles/datatable-i... Which is why it isn't really linked anywhere else.

[deleted]

You should stop insulting others and yourself.

Re: Polars: Fast DataFrame library for Rust and Python

#45

Post by ivirshup »

gpderetta wrote: post #42
abeppu wrote: post #7

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.

I believe Codd took a stab at it a few years ago. He had some success, but didn't break in data science.

The guy who coined a new term every time he had a new product to sell?

Re: Polars: Fast DataFrame library for Rust and Python

#46

Post by [deleted] »

paulgb wrote: post #29
sdfgsdf wrote: post #17

Earlier quoted context omitted.

The benchmarks speak volumes of dishonesty. They sorted the results by speed of 1st run. For a language like Julia, which is JIT-compiled, that's not a fair comparison, considering that you compile once and run millions of times. Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny because once you notice it those benchmarks a…

> considering that you compile once and run millions of times. If you’re writing data pipelines then yes, but a lot of Pandas users use it interactivity. As much as I’d rather use Julia, the last time I tried it I found myself waiting for computation far more often than with a Jupyter/Python workflow.

[deleted]

Re: Polars: Fast DataFrame library for Rust and Python

#48

Post by carterschonwald »

devin-petersohn wrote: post #30
abeppu wrote: post #7

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.

This was my PhD focus. We identified a core "dataframe algebra"[1] that encompasses all of pandas (and R/S data.frames): a total of 16 operators that cover all 600+ operators of pandas. What you describe was exactly our aim. It turns out there are a lot of operators that are really easy to support and make fast, and that gets you about 60% or so of the way to supporting all of pandas. Then there are really complex op…

This is really cool! Thx for sharing
Post reply on HN