Polars: Fast DataFrame library for Rust and Python

#1

Post by daureg » Thu, Dec 16, 2021, 10:11 PM UTC

Polars: Fast DataFrame library for Rust and Python
pola.rs

Re: Polars: Fast DataFrame library for Rust and Python

#2

Post by nas » Thu, Dec 16, 2021, 11:04 PM UTC

It looks interesting but phrases like "embarrassingly parallel execution" make my marketing hype detectors trigger. Maybe they could tone down their self promotion just a touch. Also "Even though Polars is completely written in Rust (no runtime overhead!) ...". I find that hard to believe.

Re: Polars: Fast DataFrame library for Rust and Python

#3

Post by nojito » Thu, Dec 16, 2021, 11:05 PM UTC

nas wrote: post #2

It looks interesting but phrases like "embarrassingly parallel execution" make my marketing hype detectors trigger. Maybe they could tone down their self promotion just a touch. Also "Even though Polars is completely written in Rust (no runtime overhead!) ...". I find that hard to believe.

Why?

The benchmarks speak volumes.

https://h2oai.github.io/db-benchmark/

Re: Polars: Fast DataFrame library for Rust and Python

#4

Post by ahurmazda » Thu, Dec 16, 2021, 11:10 PM UTC

I’ve been using it for the past quarter. In addition to the speed, I’m very pleased with the pyspark-esque api. This means migrating code from research to production is that much easier.

Re: Polars: Fast DataFrame library for Rust and Python

#5

Post by jmakov » Thu, Dec 16, 2021, 11:11 PM UTC

How does compare to Vaex?

Re: Polars: Fast DataFrame library for Rust and Python

#6

Post by VHRanger » Thu, Dec 16, 2021, 11:21 PM UTC

jmakov wrote: post #5

How does compare to Vaex?

That's the real question

Re: Polars: Fast DataFrame library for Rust and Python

#7

Post by abeppu » Thu, Dec 16, 2021, 11:25 PM UTC

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.

Re: Polars: Fast DataFrame library for Rust and Python

#8

Post by ddavis » Thu, Dec 16, 2021, 11:26 PM UTC

abeppu wrote: post #7

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.

There is an effort for this: https://github.com/data-apis/dataframe-api

Re: Polars: Fast DataFrame library for Rust and Python

#9

Post by space_rock » Thu, Dec 16, 2021, 11:35 PM UTC

nas wrote: post #2

It looks interesting but phrases like "embarrassingly parallel execution" make my marketing hype detectors trigger. Maybe they could tone down their self promotion just a touch. Also "Even though Polars is completely written in Rust (no runtime overhead!) ...". I find that hard to believe.

He is basically describing benefits of the rest language so it's perfectly credible

Re: Polars: Fast DataFrame library for Rust and Python

#10

Post by lern_too_spel » Thu, Dec 16, 2021, 11:47 PM UTC

nas wrote: post #2

It looks interesting but phrases like "embarrassingly parallel execution" make my marketing hype detectors trigger. Maybe they could tone down their self promotion just a touch. Also "Even though Polars is completely written in Rust (no runtime overhead!) ...". I find that hard to believe.

"Embarrassingly parallel" is a technical term, not a marketing term. https://en.wikipedia.org/wiki/Embarrassingly_parallel