Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polars Cloud and Distributed Polars now available

pola.rs

Post reply on HN
Print view View original

51–60 of 95 posts

Re: Polars Cloud and Distributed Polars now available

#51

Post by sureglymop »

I recently had to create a reproducible version of incredibly complicated and messy R concoctions our data scientists came up with.

I did it with pandas without much experience with it and a lot of AI help (essentially to fill in the blanks the data scientists had left, because they only had to do the calculation once).

I then created a polars version which uses lazyframes. It ended up being about 20x faster than the first version. I did try to do some optimizations by hand to make the execution planner work even better which I believe paid off.

If you have to do a large non interactive analytical calculation (i.e. not in a notebook) polars seems to be way ahead imo!

I do wish that it was just as easy to use as a rust library though.. the focus however seems to be on being competitive in python land mainly.

Re: Polars Cloud and Distributed Polars now available

#52

Post by tomtom1337 »

sureglymop wrote: post #51

I recently had to create a reproducible version of incredibly complicated and messy R concoctions our data scientists came up with. I did it with pandas without much experience with it and a lot of AI help (essentially to fill in the blanks the data scientists had left, because they only had to do the calculation once). I then created a polars version which uses lazyframes. It ended up being about 20x faster than the…

Out of curiosity, what makes a rust library easier to use? Could you expand on that?

Re: Polars Cloud and Distributed Polars now available

#54

Post by mrtimo »

drej wrote: post #23

Having done a bit of data engineering in my day, I'm growing more and more allergic to the DataFrame API (which I used 24/7 for years). From what I've seen over the past ~10 years, 90+% of use cases would be better served by SQL, both from the development perspective as well as debugging, onboarding, sharing, migrating etc. Give an analyst AWS Athena, DuckDB, Snowflake, whatever, and they won't have to worry about lo…

I agree with this 100%. The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here [1].

I've been using Malloy [2], which compiles to SQL (like Typescript compiles to Javascript), so instead of editing a 1000 line SQL script, it's only 18 lines of Malloy.

I'd love to see a blog post comparing a pandas approach to cleaning to an SQL/Malloy approach.

[1] https://www.youtube.com/watch?v=PFUZlNQIndo [2] https://www.malloydata.dev/

Re: Polars Cloud and Distributed Polars now available

#55

Post by ritchie46 »

tomtom1337 wrote: post #52
sureglymop wrote: post #51

I recently had to create a reproducible version of incredibly complicated and messy R concoctions our data scientists came up with. I did it with pandas without much experience with it and a lot of AI help (essentially to fill in the blanks the data scientists had left, because they only had to do the calculation once). I then created a polars version which uses lazyframes. It ended up being about 20x faster than the…

Out of curiosity, what makes a rust library easier to use? Could you expand on that?

He means that he wants our Rust library as easy as our Python lib. Which I understand as our focus has been mostly on Python.

It is where most of our userbase is and it is very hard for us to have a stable Rust API as we have a lot of internal moving parts which Rust users typically want access to (as they like to be closer to the metal), but has no stability guarantees from us.

In python, we are able to abstract and provide a stable API.

Re: Polars Cloud and Distributed Polars now available

#56

Post by gigatexal »

drej wrote: post #46
gigatexal wrote: post #45

Earlier quoted context omitted.

Again the issue you’re having is the skill level of the audience you keep bringing up not the tool.

I find it much more beneficial to lower the barrier for entry (oftentimes without any sacrifices) instead of spending time and money on upskilling everyone, just because I like engineering.

Right but nobody is saying polars or data frames is to replace SQL or is even for the masses. It’s a tool for skilled folks. I personally think the api makes sense but SQL is easier to pick up. Use whatever tools work best.

But coming into such a discussion dunking on a tool cuz it’s not for the masses makes no sense.

Re: Polars Cloud and Distributed Polars now available

#57

Post by ritchie46 »

ayhanfuat wrote: post #26
ritchie46 wrote: post #14

Earlier quoted context omitted.

I am not an expert on Spark RDDs, but AFAIK they are a more low-level data structure that offer resilience and a lower level map-reduce API. Polars Cloud maps the Polars API/DSL to distributed compute. This is more akin to Spark's high level DataFrame API. With regard to implementation, we create stages that run parts of Polars IR (internal representation) on our OSS streaming engine. Those stages run on 1 or many wo…

Can you tell a little about the status of Iceberg write support? Partitioning, maintenance etc.

We have full iceberg read support. We have done some preliminary work for iceberg write support. I think we will ship that once we have decided which Catalog we will add. The iceberg write API is intertwined with that.

Re: Polars Cloud and Distributed Polars now available

#58

Post by drej »

gigatexal wrote: post #56
drej wrote: post #46

Earlier quoted context omitted.

I find it much more beneficial to lower the barrier for entry (oftentimes without any sacrifices) instead of spending time and money on upskilling everyone, just because I like engineering.

Right but nobody is saying polars or data frames is to replace SQL or is even for the masses. It’s a tool for skilled folks. I personally think the api makes sense but SQL is easier to pick up. Use whatever tools work best. But coming into such a discussion dunking on a tool cuz it’s not for the masses makes no sense.

Read my posts again, I'm not complaining it's not for the masses, I know it isn't. I'm complaining that it's being forced upon people when there are simpler alternatives that help people focus on business problems rather than setting up virtual environments.

So I'm very much advocating for people to "[u]se whatever tools work best".

(That is - now I'm doing this. In the past I taught a course on pandas data analytics and spoke at a few PyData conferences and meetups, partly about dataframes and how useful they are. So I'm very much guilty of what all of the above.)

Re: Polars Cloud and Distributed Polars now available

#59

Post by lucasyvas »

ritchie46 wrote: post #55
tomtom1337 wrote: post #52

Earlier quoted context omitted.

Out of curiosity, what makes a rust library easier to use? Could you expand on that?

He means that he wants our Rust library as easy as our Python lib. Which I understand as our focus has been mostly on Python. It is where most of our userbase is and it is very hard for us to have a stable Rust API as we have a lot of internal moving parts which Rust users typically want access to (as they like to be closer to the metal), but has no stability guarantees from us. In python, we are able to abstract and…

I understand the user pool comment but don’t understand why you wouldn’t be able to have a rust layer that’s the same as the Python one API-wise.

I say this as a user of neither - just that I don’t see any inherent validity to that statement.

If you are saying Rust consumers want something lower level than you’re willing to make stable, just give them a higher level one and tell them to be happy with it because it matches your design philosophy.

Re: Polars Cloud and Distributed Polars now available

#60

Post by esafak »

mrtimo wrote: post #54
drej wrote: post #23

Having done a bit of data engineering in my day, I'm growing more and more allergic to the DataFrame API (which I used 24/7 for years). From what I've seen over the past ~10 years, 90+% of use cases would be better served by SQL, both from the development perspective as well as debugging, onboarding, sharing, migrating etc. Give an analyst AWS Athena, DuckDB, Snowflake, whatever, and they won't have to worry about lo…

I agree with this 100%. The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here [1]. I've been using Malloy [2], which compiles to SQL (like Typescript compiles to Javascript), so instead of editing a 1000 line SQL script, it's only 18 lines of Malloy. I'd love to see a blog post comparing a pandas approach to clea…

Have you used Malloy in a pipeline, e.g., with Airflow? If so, how was the experience?
Post reply on HN