Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polars Cloud and Distributed Polars now available

pola.rs

Post reply on HN
Print view View original

11–20 of 95 posts

Re: Polars Cloud and Distributed Polars now available

#11

Post by ritchie46 »

gigatexal wrote: post #9

Cool. But abstract away the infra knowledge to the actual instance types. Instead I’d expect the polars cloud abstraction to find me the most cost effective (spot instance) that meets my cpu and memory reqs and disk reqs. Why do I have to give it — looking at the example — the AWS instance type?

You don't have to. Passing cpu and memory works as well.

    pc.ComputeContext{
        cpus=4, 
        memory=16
    }
We are working on a minimal cluster and auto-scaling based on the query.

Re: Polars Cloud and Distributed Polars now available

#14

Post by ritchie46 »

cbb330 wrote: post #10

can you dive a bit deeper into the comparison with spark rdd

I am not an expert on Spark RDDs, but AFAIK they are a more low-level data structure that offer resilience and a lower level map-reduce API.

Polars Cloud maps the Polars API/DSL to distributed compute. This is more akin to Spark's high level DataFrame API.

With regard to implementation, we create stages that run parts of Polars IR (internal representation) on our OSS streaming engine. Those stages run on 1 or many workers create data that will be shuffled in between stages. The scheduler is responsible for creating the distributed query plan and work distribution.

Re: Polars Cloud and Distributed Polars now available

#15

Post by gigatexal »

ritchie46 wrote: post #11
gigatexal wrote: post #9

Cool. But abstract away the infra knowledge to the actual instance types. Instead I’d expect the polars cloud abstraction to find me the most cost effective (spot instance) that meets my cpu and memory reqs and disk reqs. Why do I have to give it — looking at the example — the AWS instance type?

You don't have to. Passing cpu and memory works as well. pc.ComputeContext{ cpus=4, memory=16 } We are working on a minimal cluster and auto-scaling based on the query.

Nice!

Ritchie, curious you mentioned in other responses that the SQL context stuff is out of scope for now. But I thought the SQL things were basically syntactic sugar to the dataframes in other words they both “compile” down to the same thing. If true then being able to run arbitrary SQL queries should be doable out of the box?

Re: Polars Cloud and Distributed Polars now available

#18

Post by ritchie46 »

gigatexal wrote: post #15
ritchie46 wrote: post #11

Earlier quoted context omitted.

You don't have to. Passing cpu and memory works as well. pc.ComputeContext{ cpus=4, memory=16 } We are working on a minimal cluster and auto-scaling based on the query.

Nice! Ritchie, curious you mentioned in other responses that the SQL context stuff is out of scope for now. But I thought the SQL things were basically syntactic sugar to the dataframes in other words they both “compile” down to the same thing. If true then being able to run arbitrary SQL queries should be doable out of the box?

Not right now. Our current SQLContext locally inspects schema's to convert the SQL to Polars LazyFrames (DSL).

However, this should happend during IR-resolving. E.g. the SQL should translate directly to Polars IR, and not LazyFrames. That way we can inspect/resolve all schema's server-side.

It requires a rewrite of our SQL translation in OSS. This should not be too hard, but it is quite some work. Work we eventually get to.

Re: Polars Cloud and Distributed Polars now available

#19

Post by lvl155 »

Polars is certainly better than pandas doing things locally. But that is a low bar. I’ve not had great experience using Polars on large enough datasets. I almost always end up using duckdb. If I am using SQL at the end of the day, why bother starting with Polars? With AI these days, it’s ridiculously fast to put together performant SQLs. Heck you can even make your own grammar and be done with it.

Re: Polars Cloud and Distributed Polars now available

#20

Post by whyever »

robertkoss wrote: post #12

Love it! Still don't get why one of the biggest player in the space, Databricks is overinvesting in Spark. For startups, Polars or DuckDB are completely sufficient. Other companies like Palantir already support bring your own compute.

That's a good question! Especially after Frank McSherry's COST paper [1], it's hard to imagine where the sweet spot for Spark is. I guess for Databricks it makes sense to push Spark, since they are the ones who created it. In a way, it's their competitive advantage.

[1]: https://www.usenix.org/system/files/conference/hotos15/hotos...

Post reply on HN