Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polars Cloud and Distributed Polars now available

pola.rs

Post reply on HN
Print view View original

71–80 of 95 posts

Re: Polars Cloud and Distributed Polars now available

#71

Post by orochimaaru »

drej wrote: post #37
drej wrote: post #23

Having done a bit of data engineering in my day, I'm growing more and more allergic to the DataFrame API (which I used 24/7 for years). From what I've seen over the past ~10 years, 90+% of use cases would be better served by SQL, both from the development perspective as well as debugging, onboarding, sharing, migrating etc. Give an analyst AWS Athena, DuckDB, Snowflake, whatever, and they won't have to worry about lo…

Fun aside - I actually used polars for a bit - first time I tried it, I actually thought it was broken, because it finished processing so quickly I thought it silently exited or something. So I'm definitely a fan, IF you need the DataFrame API. My point was that most people don't need it and it's oftentimes standing in the way. That's all.

Polars is very nice. I’ve used it off and on. The option to write rust udf’s for performance, easy integration of rust with Python with pyo3 will make it a real contender.

Yes, I know spark and scala exist. I use it. But the underlying Java engines and the tacky Python gateway impact performance and capacity usage. Having your primary processing engine in the same process compiled natively always helps.

Re: Polars Cloud and Distributed Polars now available

#72

Post by tomtom1337 »

ritchie46 wrote: post #55
tomtom1337 wrote: post #52

Earlier quoted context omitted.

Out of curiosity, what makes a rust library easier to use? Could you expand on that?

He means that he wants our Rust library as easy as our Python lib. Which I understand as our focus has been mostly on Python. It is where most of our userbase is and it is very hard for us to have a stable Rust API as we have a lot of internal moving parts which Rust users typically want access to (as they like to be closer to the metal), but has no stability guarantees from us. In python, we are able to abstract and…

Ah, of course. Slightly ambiguous English tricked me there. Thank you Ritchie!

Re: Polars Cloud and Distributed Polars now available

#74

Post by sureglymop »

tomtom1337 wrote: post #72
ritchie46 wrote: post #55

Earlier quoted context omitted.

He means that he wants our Rust library as easy as our Python lib. Which I understand as our focus has been mostly on Python. It is where most of our userbase is and it is very hard for us to have a stable Rust API as we have a lot of internal moving parts which Rust users typically want access to (as they like to be closer to the metal), but has no stability guarantees from us. In python, we are able to abstract and…

Ah, of course. Slightly ambiguous English tricked me there. Thank you Ritchie!

I apologize for that, English isn't my first language. Glad it was explained so well!

Re: Polars Cloud and Distributed Polars now available

#75

Post by mrtimo »

willvarfar wrote: post #70
fumeux_fume wrote: post #67

Earlier quoted context omitted.

In the same talk, Mark acknowledges that "for data science workflows, database systems are frustrating and slow." Granted DuckDB is an attempt to fix that, most data scientists don't get to choose what database the data is stored in.

(I use duckdb to query data stored in parquet files)

Same. But, I use Malloy which uses duckdb to query data stored in hundreds of parquet files (as if they were one big file).

Re: Polars Cloud and Distributed Polars now available

#76

Post by entropicdrifter »

orlp wrote: post #68
mrtimo wrote: post #54

Earlier quoted context omitted.

I agree with this 100%. The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here [1]. I've been using Malloy [2], which compiles to SQL (like Typescript compiles to Javascript), so instead of editing a 1000 line SQL script, it's only 18 lines of Malloy. I'd love to see a blog post comparing a pandas approach to clea…

> The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here. That's pandas. Polars builds on much of the same 50 years of progress in database research by offering a lazy DataFrame API which does query optimization, morsel-based columnar execution, predicate pushdown into file I/O, etc, etc. Disclaimer: I work for Po…

Just wanted to say I'm a huge fan of your work. Been using Polars for my team's main project for years and it just keeps getting better.

Re: Polars Cloud and Distributed Polars now available

#78

Post by quodlibetor »

bobbylarrybobby wrote: post #66
lucasyvas wrote: post #59

Earlier quoted context omitted.

I understand the user pool comment but don’t understand why you wouldn’t be able to have a rust layer that’s the same as the Python one API-wise. I say this as a user of neither - just that I don’t see any inherent validity to that statement. If you are saying Rust consumers want something lower level than you’re willing to make stable, just give them a higher level one and tell them to be happy with it because it ma…

The issue with Rust is that as a strict language with no function overloading (except via traits) or keyword arguments, things get very verbose. For instance, in python you can treat a string as a list of columns as in `df.select('date')` whereas in Rust you need to write `df.select([col('date')])`. Let's say you want to map a function over three columns, it's going to look something like this: ``` df.with_column( ma…

> Not much polars can do about that in Rust

I'm ignorant about the exact situation in Polars, but it seems like this is the same problem that web frameworks have to handle to enable registering arbitrary functions, and they generally do it with a FromRequest trait and macros that implement it for functions of up to N arguments. I'm curious if there are were attempts that failed for something like FromDataframe to enable at least |c: Col("a"), c2: Col("b")| {...}

https://github.com/tokio-rs/axum/blob/86868de80e0b3716d9ef39...

https://github.com/tokio-rs/axum/blob/86868de80e0b3716d9ef39...

Re: Polars Cloud and Distributed Polars now available

#79

Post by bobbylarrybobby »

quodlibetor wrote: post #78
bobbylarrybobby wrote: post #66

Earlier quoted context omitted.

The issue with Rust is that as a strict language with no function overloading (except via traits) or keyword arguments, things get very verbose. For instance, in python you can treat a string as a list of columns as in `df.select('date')` whereas in Rust you need to write `df.select([col('date')])`. Let's say you want to map a function over three columns, it's going to look something like this: ``` df.with_column( ma…

> Not much polars can do about that in Rust I'm ignorant about the exact situation in Polars, but it seems like this is the same problem that web frameworks have to handle to enable registering arbitrary functions, and they generally do it with a FromRequest trait and macros that implement it for functions of up to N arguments. I'm curious if there are were attempts that failed for something like FromDataframe to ena…

You'd still have problems.

1. There are no variadic functions so you need to take a tuple: `|(Col("a"), Col("b"))|`

2. Turbofish! `|(Col::("a"), Col::("b"))|`. This is already getting quite verbose.

3. This needs to be general over all expressions (such as `col("a").str.to_lowercase()`, `col("b") * 2`, etc), so while you could pass a type such as Col if it were IntoExpr, its conversion into an expression would immediately drop the generic type information because Expr doesn't store that (at least not in a generic parameter; the type of the underlying series is always discovered at runtime). So you can't really skip those `.i32()?` calls.

Polars definitely made the right choice here — if Expr had a generic parameter, then you couldn't store Expr of different output types in arrays because they wouldn't all have the same type. You'd have to use tuples, which would lead to abysmal ergonomics compared to a Vec (can't append or remove without a macro; need a macro to implement functions for tuples up to length N for some gargantuan N). In addition to the ergonomics, Rust’s monomorphization would make compile times absolutely explode if every combination of input Exprs’ dtypes required compiling a separate version of each function, such as `with_columns()`, which currently is only compiled separately for different container types.

The reason web frameworks can do this is because of `$( $ty: FromRequestParts + Send, )*`. All of the tuple elements share the generic parameter `S`, which would not be the case in Polars — or, if it were, would make `map` too limited to be useful.

Re: Polars Cloud and Distributed Polars now available

#80

Post by phailhaus »

orlp wrote: post #68
mrtimo wrote: post #54

Earlier quoted context omitted.

I agree with this 100%. The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here [1]. I've been using Malloy [2], which compiles to SQL (like Typescript compiles to Javascript), so instead of editing a 1000 line SQL script, it's only 18 lines of Malloy. I'd love to see a blog post comparing a pandas approach to clea…

> The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here. That's pandas. Polars builds on much of the same 50 years of progress in database research by offering a lazy DataFrame API which does query optimization, morsel-based columnar execution, predicate pushdown into file I/O, etc, etc. Disclaimer: I work for Po…

The DataFrame interface itself is the problem. It's incredibly hard to read, write, debug, and test. Too much work has gone into reducing keystrokes rather than developing a better tool.
Post reply on HN