Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polars Cloud and Distributed Polars now available

pola.rs

Post reply on HN
Print view View original

81–90 of 95 posts

Re: Polars Cloud and Distributed Polars now available

#81

Post by riku_iki »

drej wrote: post #28
robertkoss wrote: post #24

Earlier quoted context omitted.

That is a false dichotomy. You can use SQL tools but still have to choose the instance type. Especially when considering testability and composability, using a DataFrame API inside regular languages like Python is far superior IMO.

As a user? No, I don't have to choose. What I'm saying is that analysts (who this Polars Cloud targets, just like Coiled or Databricks) shouldn't worry about instance types, shuffling performance, join strategies, JVM versions, cross-AZ pricing etc. In most cases, they should just get a connection string and/or a web UI to run their queries, everything abstracted from them. Sure, Python code is more testable and comp…

> analysts (who this Polars Cloud targets, just like Coiled or Databricks) shouldn't worry about instance types, shuffling performance, join strategies,

I think this part(query optimizations) in general not solved/solvable, and it is sometimes/often(depending on domain) necessary to digg into details to make data transformation working.

Re: Polars Cloud and Distributed Polars now available

#82

Post by dev_l1x_be »

phailhaus wrote: post #80
orlp wrote: post #68

Earlier quoted context omitted.

> The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here. That's pandas. Polars builds on much of the same 50 years of progress in database research by offering a lazy DataFrame API which does query optimization, morsel-based columnar execution, predicate pushdown into file I/O, etc, etc. Disclaimer: I work for Po…

The DataFrame interface itself is the problem. It's incredibly hard to read, write, debug, and test. Too much work has gone into reducing keystrokes rather than developing a better tool.

Not sure what you mean by this. The table concept is the same age as computers. Here is a table, do something with it -> this is the high level df api. All the functions make sense, what is hard to read, write or debug here?

I have used Polars to process 600M of xml files (with a bit of a hack) and the polars part of the code is readable with minimal comments.

Polars has a better api than pandas, at least the intent is easier to understand. (lazyness, yay)

Re: Polars Cloud and Distributed Polars now available

#84

Post by phailhaus »

dev_l1x_be wrote: post #82
phailhaus wrote: post #80

Earlier quoted context omitted.

The DataFrame interface itself is the problem. It's incredibly hard to read, write, debug, and test. Too much work has gone into reducing keystrokes rather than developing a better tool.

Not sure what you mean by this. The table concept is the same age as computers. Here is a table, do something with it -> this is the high level df api. All the functions make sense, what is hard to read, write or debug here? I have used Polars to process 600M of xml files (with a bit of a hack) and the polars part of the code is readable with minimal comments. Polars has a better api than pandas, at least the intent…

The problem with the dataframe API is that whenever you want to change a small part of your logic, you usually have to rethink and rewrite the whole solution. It is too difficult to write reusable code. Too many functions that try to do too many things with a million kwargs that each have their own nuances. This is because these libraries tend to favor fewer keystrokes over composable design. So the easy stuff is easy and makes for pretty docs, but the hard stuff is obnoxious to reason through.

This article explains it pretty well: https://dynomight.net/numpy/

Re: Polars Cloud and Distributed Polars now available

#86

Post by orlp »

phailhaus wrote: post #84
dev_l1x_be wrote: post #82

Earlier quoted context omitted.

Not sure what you mean by this. The table concept is the same age as computers. Here is a table, do something with it -> this is the high level df api. All the functions make sense, what is hard to read, write or debug here? I have used Polars to process 600M of xml files (with a bit of a hack) and the polars part of the code is readable with minimal comments. Polars has a better api than pandas, at least the intent…

The problem with the dataframe API is that whenever you want to change a small part of your logic, you usually have to rethink and rewrite the whole solution. It is too difficult to write reusable code. Too many functions that try to do too many things with a million kwargs that each have their own nuances. This is because these libraries tend to favor fewer keystrokes over composable design. So the easy stuff is eas…

With all due respect, have you actually used the Polars expression API? We actually strive for composability of simple functions over dedicated methods with tons of options, where possible.

The original comment I responded to was confusing Pandas with Polars, and now your blog post refers to Numpy, but Polars takes a completely different approach to dataframes/data processing than either of these tools.

Re: Polars Cloud and Distributed Polars now available

#87

Post by [deleted] »

phailhaus wrote: post #80
orlp wrote: post #68

Earlier quoted context omitted.

> The creator of duckdb argues that people using pandas are missing out of the 50 years of progress in database research, in the first 5 minutes of his talk here. That's pandas. Polars builds on much of the same 50 years of progress in database research by offering a lazy DataFrame API which does query optimization, morsel-based columnar execution, predicate pushdown into file I/O, etc, etc. Disclaimer: I work for Po…

The DataFrame interface itself is the problem. It's incredibly hard to read, write, debug, and test. Too much work has gone into reducing keystrokes rather than developing a better tool.

[deleted]

Re: Polars Cloud and Distributed Polars now available

#88

Post by closed »

phailhaus wrote: post #84
dev_l1x_be wrote: post #82

Earlier quoted context omitted.

Not sure what you mean by this. The table concept is the same age as computers. Here is a table, do something with it -> this is the high level df api. All the functions make sense, what is hard to read, write or debug here? I have used Polars to process 600M of xml files (with a bit of a hack) and the polars part of the code is readable with minimal comments. Polars has a better api than pandas, at least the intent…

The problem with the dataframe API is that whenever you want to change a small part of your logic, you usually have to rethink and rewrite the whole solution. It is too difficult to write reusable code. Too many functions that try to do too many things with a million kwargs that each have their own nuances. This is because these libraries tend to favor fewer keystrokes over composable design. So the easy stuff is eas…

I have used numpy, but don't understand what it has to do with dataframe apis

Take two examples of dataframe apis, dplyr and ibis. Both can run on a range of SQL backends because dataframe apis are very similar to SQL DML apis.

Moreover, the SQL translation for tools for pivot_longer in R are a good illustration of complex dynamics dataframe apis can support, that you'd use something like dbt to implement in your SQL models. duckdb allows dynamic column selection in unpivot. But in some SQL dialects this is impossible. dataframe apis -> SQL tools (or dbt) enable them in these dialects.

Re: Polars Cloud and Distributed Polars now available

#89

Post by ketozhang »

drej wrote: post #23

Having done a bit of data engineering in my day, I'm growing more and more allergic to the DataFrame API (which I used 24/7 for years). From what I've seen over the past ~10 years, 90+% of use cases would be better served by SQL, both from the development perspective as well as debugging, onboarding, sharing, migrating etc. Give an analyst AWS Athena, DuckDB, Snowflake, whatever, and they won't have to worry about lo…

I think your argument focuses a lot on the scenario where you already have cleaned data (i.e., data warehouse). I and many other data engineers agree, you're better off with hosting it on SQL RDBMS.

However, before that, you need a lot of code to clean the data and raw data does not fit well into a structured RDBMS. Here you choose to either map your raw data into row view or a table view. You're now left with the choice of either inventing your own domain object (row view) or use a dataframe (table view).

Re: Polars Cloud and Distributed Polars now available

#90

Post by willvarfar »

mrtimo wrote: post #75
willvarfar wrote: post #70

Earlier quoted context omitted.

(I use duckdb to query data stored in parquet files)

Same. But, I use Malloy which uses duckdb to query data stored in hundreds of parquet files (as if they were one big file).

I haven't looked at Mallory, but I do regularly scan lots of parquet files using wildcards etc from duckdb. Its a neat builtin duckdb feature.
Post reply on HN