Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polars Cloud and Distributed Polars now available

pola.rs

Post reply on HN
Print view View original

91–95 of 95 posts

Re: Polars Cloud and Distributed Polars now available

#91

Post by doctaj »

phailhaus wrote: post #80
orlp wrote: post #68

Earlier quoted context omitted.

> The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here. That's pandas. Polars builds on much of the same 50 years of progress in database research by offering a lazy DataFrame API which does query optimization, morsel-based columnar execution, predicate pushdown into file I/O, etc, etc. Disclaimer: I work for Po…

The DataFrame interface itself is the problem. It's incredibly hard to read, write, debug, and test. Too much work has gone into reducing keystrokes rather than developing a better tool.

Assuming you’re comparing polars/data frames to sql… SQL has literally the worst debugging experience imaginable.

Re: Polars Cloud and Distributed Polars now available

#92

Post by quodlibetor »

bobbylarrybobby wrote: post #79
quodlibetor wrote: post #78

Earlier quoted context omitted.

> Not much polars can do about that in Rust I'm ignorant about the exact situation in Polars, but it seems like this is the same problem that web frameworks have to handle to enable registering arbitrary functions, and they generally do it with a FromRequest trait and macros that implement it for functions of up to N arguments. I'm curious if there are were attempts that failed for something like FromDataframe to ena…

You'd still have problems. 1. There are no variadic functions so you need to take a tuple: `|(Col ("a"), Col ("b"))|` 2. Turbofish! `|(Col:: ("a"), Col:: ("b"))|`. This is already getting quite verbose. 3. This needs to be general over all expressions (such as `col("a").str.to_lowercase()`, `col("b") * 2`, etc), so while you could pass a type such as Col if it were IntoExpr, its conversion into an expression would im…

Thanks for the insight!

Re: Polars Cloud and Distributed Polars now available

#93

Post by jkkola »

I absolutely love Polars. I work on some unholy dirty data and the ease of use, chaining, speed are a godsend. One dataset that previously took 40 minutes in Pandas now takes two minutes in Polars. Granted, the Pandas query could be optimized, but out of the box, Polars eats pandas when it comes to speed and efficiency.

I basically ditched SQL for most of my analytical work because it's way easier to understand for my juniors (we're not technically a tech team) so it's a total win in my eyes.

Re: Polars Cloud and Distributed Polars now available

#94

Post by yencabulator »

ohnoesjmr wrote: post #85

I guess could be a good contender for replacing spark, however, I suspect the fact spark is free and open source, which forms a community around it, means that dpolars might struggle to gain traction, when it's gated by a credit card.

I would expect a better contender for Spark to be something that's actually open source, such as https://github.com/apache/datafusion-ballista

Re: Polars Cloud and Distributed Polars now available

#95

Post by yencabulator »

Arnechos wrote: post #29

How does it relate to Apache DataFusion/Ballista?

It's an open-core competitor that started more from the "DataFrames for Python" end of the spectrum, where DataFusion went pretty strong into "we can handle SQL" (while still having a dataframes API).
Post reply on HN