Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polars: Fast DataFrame library for Rust and Python

pola.rs

Post reply on HN
Print view View original

51–60 of 131 posts

Re: Polars: Fast DataFrame library for Rust and Python

#51

Post by cigrainger »

civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

I've been working on a dataframe library for Elixir that's built on top of Polars and that's heavily influenced by dplyr if you're interested in checking it out: https://github.com/elixir-nx/explorer

Re: Polars: Fast DataFrame library for Rust and Python

#53

Post by queuebert »

paulgb wrote: post #29
sdfgsdf wrote: post #17

Earlier quoted context omitted.

The benchmarks speak volumes of dishonesty. They sorted the results by speed of 1st run. For a language like Julia, which is JIT-compiled, that's not a fair comparison, considering that you compile once and run millions of times. Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny because once you notice it those benchmarks a…

> considering that you compile once and run millions of times. If you’re writing data pipelines then yes, but a lot of Pandas users use it interactivity. As much as I’d rather use Julia, the last time I tried it I found myself waiting for computation far more often than with a Jupyter/Python workflow.

Give it another try. They've improved the first run times quite a bit over the last few versions. Package precompilation has gotten way better as well.

Re: Polars: Fast DataFrame library for Rust and Python

#55

Post by paulgb »

queuebert wrote: post #53
paulgb wrote: post #29

Earlier quoted context omitted.

> considering that you compile once and run millions of times. If you’re writing data pipelines then yes, but a lot of Pandas users use it interactivity. As much as I’d rather use Julia, the last time I tried it I found myself waiting for computation far more often than with a Jupyter/Python workflow.

Give it another try. They've improved the first run times quite a bit over the last few versions. Package precompilation has gotten way better as well.

Glad to hear it, I will!

Re: Polars: Fast DataFrame library for Rust and Python

#56

Post by vatican_banker »

vavooom wrote: post #26
civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

Also worth plugging the advanced speed of R’s data.table package which continues to trump dplyr to this day. The syntax is also more compact and straightforward once you understand how to query data with it.

In what way data.table trumps dplyr? Genuinely interested in knowing.

While data.table is faster than dplyr, data manipulations with data.table are difficult to read/understand/maintain.

dplyr also grew into a full-fledge list of libraries to work on data-related projects (the tidyverse). These libraries are _very_ well thought out and enables productivity with minimal learning curve [anecdotal]

Re: Polars: Fast DataFrame library for Rust and Python

#57

Post by cardosof »

devin-petersohn wrote: post #30
abeppu wrote: post #7

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.

This was my PhD focus. We identified a core "dataframe algebra"[1] that encompasses all of pandas (and R/S data.frames): a total of 16 operators that cover all 600+ operators of pandas. What you describe was exactly our aim. It turns out there are a lot of operators that are really easy to support and make fast, and that gets you about 60% or so of the way to supporting all of pandas. Then there are really complex op…

Awesome work, thanks!

Re: Polars: Fast DataFrame library for Rust and Python

#58

Post by adgjlsfhk1 »

queuebert wrote: post #53
paulgb wrote: post #29

Earlier quoted context omitted.

> considering that you compile once and run millions of times. If you’re writing data pipelines then yes, but a lot of Pandas users use it interactivity. As much as I’d rather use Julia, the last time I tried it I found myself waiting for computation far more often than with a Jupyter/Python workflow.

Give it another try. They've improved the first run times quite a bit over the last few versions. Package precompilation has gotten way better as well.

DataFrames1.3 is a lot faster specifically.

Re: Polars: Fast DataFrame library for Rust and Python

#60

Post by adgjlsfhk1 »

nojito wrote: post #36
sdfgsdf wrote: post #17

Earlier quoted context omitted.

The benchmarks speak volumes of dishonesty. They sorted the results by speed of 1st run. For a language like Julia, which is JIT-compiled, that's not a fair comparison, considering that you compile once and run millions of times. Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny because once you notice it those benchmarks a…

>The benchmarks speak volumes of dishonesty. Not really. They are designed to showcase a common use case across multiple technologies. The beauty of this benchmark is that there is a hardware limit included so that it forces you to create novel solutions to perform well. >Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny be…

the benchmarks are a bit out of date (missing DataFrames 1.2/1.3, Julia 1.7, CSV 0.9). I'm planning on running an updated version this weekend.
Post reply on HN