Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Polars Cheatsheet (based on our O'Reilly book)

opensource.posit.co

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Python Polars Cheatsheet (based on our O'Reilly book)

#31

Post by ForceBru »

paulfharrison wrote: post #23

I'm sure Polars is great, but I can't get over needing 10 characters of ceremony every time I want to refer to a column in a data frame. pl.col("...")

Sure, but these ten characters let you treat columns as values and do math on them, which is super intuitive, in my opinion. I've been using Pandas for quite some time and always kinda sucked at it. One day I decided to give this new library Polars a try. Now I can do things I couldn't even dream of with Pandas! And it's fast, too!

I think of `pl.col` as delayed evaluation: I want to do math on the vector of values of this column. But wait, let me just refer to the name of that column and build the expression that I want to compute. Then I hand this expression to Polars and it retrieves the actual values of the columns my expression refers to and executes the operations.

IMO, it would've been great to just do math on strings, like `"Amount" * "Price" - "Losses"`, but programming languages either don't allow math on strings or that math is actually string concatenation, which is not what we want. So we have to wrap the name of the column into some object. This is just an API thing.

As a side note, it's such a pity that there's basically no Polars for the Julia language! There is some wrapper package, but it seems old and unmaintained. I can't seem to properly learn DataFrames.jl for some reason, I always miss Polars when I use Julia.

Re: Python Polars Cheatsheet (based on our O'Reilly book)

#33

Post by laGrenouille »

thijsn wrote: post #25
paulfharrison wrote: post #23

I'm sure Polars is great, but I can't get over needing 10 characters of ceremony every time I want to refer to a column in a data frame. pl.col("...")

I've seen people with exactly that frustration use "import polars.col as c" and use c("colname") instead!

Yes, this (as the even shorter c.colname) and the fact that you can do var= in place of assign in with_columns/agg changed my whole outlook on polars. Have been using it as my main driver for the past year.

Re: Python Polars Cheatsheet (based on our O'Reilly book)

#34

Post by mmplxx »

thijsn wrote: post #25
paulfharrison wrote: post #23

I'm sure Polars is great, but I can't get over needing 10 characters of ceremony every time I want to refer to a column in a data frame. pl.col("...")

I've seen people with exactly that frustration use "import polars.col as c" and use c("colname") instead!

Or c.colname

Re: Python Polars Cheatsheet (based on our O'Reilly book)

#35

Post by nojito »

mrtimo wrote: post #15

I've moved from python/polars/pandas to DuckDB and have not looked back

SQL is a terrible api for data transformation.

https://duckdb.org/docs/lts/sql/statements/pivot#limitations

vs

https://docs.pola.rs/api/python/stable/reference/dataframe/a...

Re: Python Polars Cheatsheet (based on our O'Reilly book)

#36

Post by paulfharrison »

mmplxx wrote: post #32
paulfharrison wrote: post #23

I'm sure Polars is great, but I can't get over needing 10 characters of ceremony every time I want to refer to a column in a data frame. pl.col("...")

c = pl.col c.foo + c.bar

Nice. This is close enough to an R level of brevity, without any R dark magic.

Would be a great addition to the cheatsheet.

Re: Python Polars Cheatsheet (based on our O'Reilly book)

#37

Post by latent-person »

ForceBru wrote: post #31
paulfharrison wrote: post #23

I'm sure Polars is great, but I can't get over needing 10 characters of ceremony every time I want to refer to a column in a data frame. pl.col("...")

Sure, but these ten characters let you treat columns as values and do math on them, which is super intuitive, in my opinion. I've been using Pandas for quite some time and always kinda sucked at it. One day I decided to give this new library Polars a try. Now I can do things I couldn't even dream of with Pandas! And it's fast, too! I think of `pl.col` as delayed evaluation: I want to do math on the vector of values o…

You should look into dplyr [1] (part of the tidyverse) in R to see how intuitive this can get. You can do math directly on columns:

  df |> dplyr::mutate(profit = Amount * Price - Losses)
For Julia, take a look at TidierData.jl [2], which provides similar tidy syntax via macros.

[1] https://dplyr.tidyverse.org/

[2] https://tidierorg.github.io/TidierData.jl/latest/

Re: Python Polars Cheatsheet (based on our O'Reilly book)

#38

Post by dajt »

dec0dedab0de wrote: post #13
clircle wrote: post #3

I get that the data science world has moved on to python, but I always felt that R's data.table had the slickest dataframe developer experience. I have toyed with Polars for a few hours, maybe I should give it a better chance.

I first learned about R at a python user group meeting. It was when Pandas was new, and we were having a bunch of talks about it. Wes McKinney even came to give one before going to Pycon. Anyway, the general consensus at the time was that R was much nicer once you had your data, and if all you had to do was transform it. But that everything else was better in Python. One of our group members did an experimental proje…

I use rpy2 for that. The scientists wrote the calc engine in R (via my Python transscription of the Excel original :) and I wrote the dashboard and data pipeline in Python. The data pipeline batch process embeds the R interpreter to avoid spawning processes all the time.

Re: Python Polars Cheatsheet (based on our O'Reilly book)

#39

Post by rtpg »

Pandas and Polars are both a place where I _really_ would love to have some sort of macro subsystem for Python. `pl.col(...)` is a neat trick for slicing, the pandas `df[df["foo"] == "bar"]` thing has always felt a bit of a mouthful (especially if you deign to use a longer name for your dataframe).

I appreciate Polars offering some alternative APIs for poking around in data, though. I feel like at some point someone will land on a _very_ nice to use API

Re: Python Polars Cheatsheet (based on our O'Reilly book)

#40

Post by rtpg »

holub008 wrote: post #22
wsowens wrote: post #14

Despite writing most of my procedural code in Python, I've always preferred doing my data analysis in R. For all of R's warts, the ergonomics of the dplyr + ggplot + the rest of the tidyverse are very tough to beat. My few attempts to use Pandas and matplotlib/seaborne have always proved frustrating. Based on this cheatsheet though, it seems like Polars addresses some of the friction of Pandas. Looking forward to try…

Agreed, as an R and polars user. The fundamental advantage R holds over other languages/libraries is expressions. The ability to reference columns directly AND interoperate with vectorized base ops in R is unfair. Of course, this super power is equally confusing to learners, fraught for production code, etc.

do you have a snippet of what this looks like in R?
Post reply on HN