Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polars: Fast DataFrame library for Rust and Python

pola.rs

Post reply on HN
Print view View original

121–130 of 131 posts

Re: Polars: Fast DataFrame library for Rust and Python

#121

Post by austospumanto »

abeppu wrote: post #7

There are so many dataframe libraries, many of which have APIs closely following pandas, but not drop-in replacements. I wish we could agree on a standard describing the core parts of what a dataframe must do, such that code depending only on those operations can easily move between dataframes.

https://github.com/austospumanto/minimal-pandas-api-for-pola...

pip install minimal-pandas-api-for-polars

I wrote a library that wraps polars DataFrame and Series objects to allow you to use them with the same syntax as with pandas DataFrame and Series objects. The goal is not to be a replacement for polars' objects and syntax, but rather to (1) Allow you to provide (wrapped) polars objects as arguments to existing functions in your codebase that expect pandas objects and (2) Allow you to continue writing code (especially EDA in notebooks) using the pandas syntax you know and (maybe) love while you're still learning the polars syntax, but with the underlying objects being all-polars. All methods of polars' objects are still available, allowing you to interweave pandas syntax and polars syntax when working with MppFrame and MppSeries objects.

Furthermore, the goal should always be to transition away from this library over time, as the LazyFrame optimizations offered by polars can never be fully taken advantage of when using pandas-based syntax (as far as I can tell). In the meantime, the code in this library has allowed me to transition my company's pandas-centric code to polars-centric code more quickly, which has led to significant speedups and memory savings even without being able to take full advantage of polars' lazy evaluation. To be clear, these gains have been observed both when working in notebooks in development and when deployed in production API backends / data pipelines.

I'm personally just adding methods to the MppFrame and MppSeries objects whenever I try to use pandas syntax and get AttributeErrors.

Re: Polars: Fast DataFrame library for Rust and Python

#122

Post by sdfgsdf »

ritchie46 wrote: post #82
sdfgsdf wrote: post #17

Earlier quoted context omitted.

The benchmarks speak volumes of dishonesty. They sorted the results by speed of 1st run. For a language like Julia, which is JIT-compiled, that's not a fair comparison, considering that you compile once and run millions of times. Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny because once you notice it those benchmarks a…

Note that the compile times of julia are not included in the benchmarks. If you read the website, you'd seen that the grapsh show the first (excluding the compilation) and the second run (with hot cache). Also in the second run, julia is not the fastest. Julia would not be faster than Rust, its got a garbage collector. This is what you see in the join benchmarks that really push the allocator. Next to that, the datab…

> Note that the compile times of julia are not included in the benchmarks. If you read the website, you'd seen that the grapsh show the first (excluding the compilation) and the second run (with hot cache).

Here's my view: The author of that page has commented here on HN; If my claim was so outrageously wrong as you claim, he would've corrected it.

Re: Polars: Fast DataFrame library for Rust and Python

#123

Post by optimalonpaper »

I'm reading all these comments and keep asking myself if I'm missing something, because I honestly sort of like pandas' API?

Sure dplyr is nice -- it felt that way on rare occasions that I got to use it, at least -- but you get used to anything.

So since I'm using python and know it quite well, I'm just more comfortable sticking with python's pandas framework rather than switching to R for data processing

Re: Polars: Fast DataFrame library for Rust and Python

#124

Post by FridgeSeal »

BiteCode_dev wrote: post #106
civilized wrote: post #13

In my world, anything that isn't "identical to R's dplyr API but faster" just isn't quite worth switching for. There's absolutely no contest: dplyr has the most productive API and that matters to me more than anything else. But I'm glad to see Polars moves away from the kludgey sprawl of the Pandas API towards the perfection of dplyr... while also being blazingly fast! Now just mix in a bit of DSL so people aren't ob…

You don't need to write "import pandas; pandas.bla()", you can do "from pandas import *; anything_in_pandas()" if you want quick and dirty.

And if you want you and your team mates to hate you when they need to work on your code later, and you’ve got random, mystery functions all over the place.

Re: Polars: Fast DataFrame library for Rust and Python

#125

Post by civilized »

nojito wrote: post #113
vatican_banker wrote: post #56

Earlier quoted context omitted.

In what way data.table trumps dplyr? Genuinely interested in knowing. While data.table is faster than dplyr, data manipulations with data.table are difficult to read/understand/maintain. dplyr also grew into a full-fledge list of libraries to work on data-related projects (the tidyverse). These libraries are _very_ well thought out and enables productivity with minimal learning curve [anecdotal]

The difficulty to read is a misnomer. Dt[rows, columns, groups] Assuming your dplyr code is generally split apply combine, the dt version is shorter and easier to reason around. https://atrebas.github.io/post/2019-03-03-datatable-dplyr/

I disagree. Doing data manipulation one action at a time in a piped sequence is easiest to reason about because the state right before you apply a new operation is always clear.

data.table, on the other hand, is a fancy clever gadget with many knobs and buttons you have to turn and press just so to get the desired result. It's only simple if all you do is filter, group by, and summarize.

To illustrate, let's look at what you have to do in data.table in order to achieve the equivalent of a grouped filter in dplyr (from the dtplyr translation vignette):

dplyr:

  df %>% 
    group_by(a) %>%
    filter(b
data.table:

  DT[DT[, .I[b
Compared to the simple, declarative feel of the dplyr, there's a lot of weird stuff going on in the data.table version. You have to put DT inside itself? What is .I? Where did V1 come from? Janky stuff.

(And yes I know precisely what is going on in the data.table version, I just think it's ugly and illustrates my point about composability and legibility extremely well.)

The reason data.table has all these independent knobs is because it wants you to cram your entire query into a single command, so it can optimize the query more easily and squeeze every drop of performance. NOT because it's more understandable, because it isn't.

The best of both worlds -- an optimizable query and one-action-at-a-time syntax -- can be achieved with a lazy system like Apache Spark or dtplyr.

Re: Polars: Fast DataFrame library for Rust and Python

#126

Post by fault1 »

sdfgsdf wrote: post #122
ritchie46 wrote: post #82

Earlier quoted context omitted.

Note that the compile times of julia are not included in the benchmarks. If you read the website, you'd seen that the grapsh show the first (excluding the compilation) and the second run (with hot cache). Also in the second run, julia is not the fastest. Julia would not be faster than Rust, its got a garbage collector. This is what you see in the join benchmarks that really push the allocator. Next to that, the datab…

> Note that the compile times of julia are not included in the benchmarks. If you read the website, you'd seen that the grapsh show the first (excluding the compilation) and the second run (with hot cache). Here's my view: The author of that page has commented here on HN; If my claim was so outrageously wrong as you claim, he would've corrected it.

yeah, but your claim was "Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run"

notice this isn't even a language vs language benchmark. it's libraries and frameworks.

plus I don't think even the author of the julia library in question would agree with your statement: https://discourse.julialang.org/t/the-state-of-dataframes-jl...

as mentioned in that thread, GC and strings, or especially a combination of the two, can be very much a downer in terms of julia performance. That's actually pretty surprising since strings are often as important if not more important than numbers for a lot of data processing needs.

I'd also say in terms of compilation time, some autocaching layer outside of precompilation would do wonders.

Re: Polars: Fast DataFrame library for Rust and Python

#127

Post by throwawaybutwhy »

1egg0myegg0 wrote: post #67
adgjlsfhk1 wrote: post #60

Earlier quoted context omitted.

the benchmarks are a bit out of date (missing DataFrames 1.2/1.3, Julia 1.7, CSV 0.9). I'm planning on running an updated version this weekend.

If you wouldn't mind, please update DuckDB as well!

It's obvious that you're promoting duck eggs at the expense of, say, chicken eggs or quail eggs or even ostrich eggs. Maybe you could tone that down a bit.

Re: Polars: Fast DataFrame library for Rust and Python

#128

Post by kruxigt »

paulgb wrote: post #29
sdfgsdf wrote: post #17

Earlier quoted context omitted.

The benchmarks speak volumes of dishonesty. They sorted the results by speed of 1st run. For a language like Julia, which is JIT-compiled, that's not a fair comparison, considering that you compile once and run millions of times. Note also that Julia would be number 1 in almost all of those benchmarks if you were to rank by speed of second run (as expected...). It's funny because once you notice it those benchmarks a…

> considering that you compile once and run millions of times. If you’re writing data pipelines then yes, but a lot of Pandas users use it interactivity. As much as I’d rather use Julia, the last time I tried it I found myself waiting for computation far more often than with a Jupyter/Python workflow.

Well, then it depends if interactively means redefined methods or just glueing together existing methods in new ways with new data. If it's the latter then no new compilations are needed. If it's the former then we are dealing with a type of work that can probably not be done in pure Python or similar and would require some kind of compilation anyway.

Re: Polars: Fast DataFrame library for Rust and Python

#129

Post by nojito »

civilized wrote: post #125
nojito wrote: post #113

Earlier quoted context omitted.

The difficulty to read is a misnomer. Dt[rows, columns, groups] Assuming your dplyr code is generally split apply combine, the dt version is shorter and easier to reason around. https://atrebas.github.io/post/2019-03-03-datatable-dplyr/

I disagree. Doing data manipulation one action at a time in a piped sequence is easiest to reason about because the state right before you apply a new operation is always clear. data.table, on the other hand, is a fancy clever gadget with many knobs and buttons you have to turn and press just so to get the desired result. It's only simple if all you do is filter, group by, and summarize. To illustrate, let's look at…

Your code golf example makes no sense.

    B_mean
Unlike the dplyr solution the dt solution is robust and we can independently test to make sure the mean of b makes sense.

The very easy to reason around concept of dt[rows, columns, groups] makes the code extremely clear.

Your translation example is absolutely bonkers because it’s trying to pigeonhole the simplicity of dt into the nonsense that is dplyr.

Re: Polars: Fast DataFrame library for Rust and Python

#130

Post by Hasnep »

fault1 wrote: post #120
Hasnep wrote: post #119

Earlier quoted context omitted.

The convention in Julia is that a package that defines a type Abc is called Abcs.jl. Also, DataFrames.jl provides its own manipulation functions which DataFramesMeta is a wrapper around using metaprogramming, hence the name.

That makes sense, but I still think the meta name is confusing. I mean, as a user the fact that it was implemented using metaprogramming techniques has no bearing, it's an implementation detail. Actually, my brain never thought to associate meta in this context with metaprogramming. Makes sense in hindsight, but still confusing. But still, I can't really come up with a nicer name. VerbalDataFrames to match the dplyr…

Yeah, I agree it's not a good name. I think using the word macro instead of meta is more useful to the user, something like DataFramesMacros.jl.
Post reply on HN