Polars Cloud and Distributed Polars now available

Post by gigatexal » Thu, Sep 04, 2025, 10:49 AM UTC

ritchie46 wrote: post #18

gigatexal wrote: post #15

Earlier quoted context omitted.

Nice! Ritchie, curious you mentioned in other responses that the SQL context stuff is out of scope for now. But I thought the SQL things were basically syntactic sugar to the dataframes in other words they both “compile” down to the same thing. If true then being able to run arbitrary SQL queries should be doable out of the box?

Not right now. Our current SQLContext locally inspects schema's to convert the SQL to Polars LazyFrames (DSL). However, this should happend during IR-resolving. E.g. the SQL should translate directly to Polars IR, and not LazyFrames. That way we can inspect/resolve all schema's server-side. It requires a rewrite of our SQL translation in OSS. This should not be too hard, but it is quite some work. Work we eventually…

Post by robertkoss » Thu, Sep 04, 2025, 11:01 AM UTC

drej wrote: post #23

Having done a bit of data engineering in my day, I'm growing more and more allergic to the DataFrame API (which I used 24/7 for years). From what I've seen over the past ~10 years, 90+% of use cases would be better served by SQL, both from the development perspective as well as debugging, onboarding, sharing, migrating etc. Give an analyst AWS Athena, DuckDB, Snowflake, whatever, and they won't have to worry about lo…

Post by ayhanfuat » Thu, Sep 04, 2025, 11:05 AM UTC

ritchie46 wrote: post #14

cbb330 wrote: post #10

can you dive a bit deeper into the comparison with spark rdd

I am not an expert on Spark RDDs, but AFAIK they are a more low-level data structure that offer resilience and a lower level map-reduce API. Polars Cloud maps the Polars API/DSL to distributed compute. This is more akin to Spark's high level DataFrame API. With regard to implementation, we create stages that run parts of Polars IR (internal representation) on our OSS streaming engine. Those stages run on 1 or many wo…

Post by drej » Thu, Sep 04, 2025, 11:16 AM UTC

robertkoss wrote: post #24

drej wrote: post #23

Having done a bit of data engineering in my day, I'm growing more and more allergic to the DataFrame API (which I used 24/7 for years). From what I've seen over the past ~10 years, 90+% of use cases would be better served by SQL, both from the development perspective as well as debugging, onboarding, sharing, migrating etc. Give an analyst AWS Athena, DuckDB, Snowflake, whatever, and they won't have to worry about lo…

That is a false dichotomy. You can use SQL tools but still have to choose the instance type. Especially when considering testability and composability, using a DataFrame API inside regular languages like Python is far superior IMO.

Post by boomer_joe » Thu, Sep 04, 2025, 11:20 AM UTC

ritchie46 wrote: post #27

boomer_joe wrote: post #25

I don't understand. Can I use distributed Polars with my own machines or do I have to buy cloud compute to run distributed queries (I don't want that). If not, is this planned?

On-premises is in the works. We expect this in a couple of months. Currently it is managed on AWS only.