Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Polars Cloud and Distributed Polars now available

pola.rs

Post reply on HN
Print view View original

41–50 of 95 posts

Re: Polars Cloud and Distributed Polars now available

#41

Post by infecto »

lvl155 wrote: post #19

Polars is certainly better than pandas doing things locally. But that is a low bar. I’ve not had great experience using Polars on large enough datasets. I almost always end up using duckdb. If I am using SQL at the end of the day, why bother starting with Polars? With AI these days, it’s ridiculously fast to put together performant SQLs. Heck you can even make your own grammar and be done with it.

I guess if it’s too large to be performant than SQL can be the way to go. I avoid sql for one off tasks though as I can more easily grok transformations in polars code than sql queries.

Re: Polars Cloud and Distributed Polars now available

#42

Post by dkdcio »

rorads wrote: post #7
anonu wrote: post #3

So competing with SnowFlake?

EDIT: I think the below is correct, but I’ve just seen in the main product landing page that for a certain benchmark it’s an order of magnitude cheaper AND faster than AWS glue, so that’s the target market by the looks of things. —— I don’t think so - probably more in the realms of spark and, based on the roadmap, airflow. For me it would be about doing big data analytics / dashboarding / ML or DS data prep. My under…

moreso competing with Coiled Computing (Dask version, very similar, you can run Polars there too). and then Databricks more than Snowflake, but all of these data platforms converge on similar features. also competing with Fivetran eventually after their acquisition yesterday

Re: Polars Cloud and Distributed Polars now available

#43

Post by forks »

ritzaco wrote: post #21

Maybe just me, but for anyone else who was confused - Polars (Pola.rs) - the DataFrames library that now has a cloud version - Polar (Polar.sh) - Payments and MoR service built on top of Stripe

- Polar, the authorization DSL created by Oso

It's a common name

Re: Polars Cloud and Distributed Polars now available

#44

Post by gigatexal »

robertkoss wrote: post #24
drej wrote: post #23

Having done a bit of data engineering in my day, I'm growing more and more allergic to the DataFrame API (which I used 24/7 for years). From what I've seen over the past ~10 years, 90+% of use cases would be better served by SQL, both from the development perspective as well as debugging, onboarding, sharing, migrating etc. Give an analyst AWS Athena, DuckDB, Snowflake, whatever, and they won't have to worry about lo…

That is a false dichotomy. You can use SQL tools but still have to choose the instance type. Especially when considering testability and composability, using a DataFrame API inside regular languages like Python is far superior IMO.

Yeah it makes no sense.

Why is the dataframe approach getting hate when you’re talking about runtime details?

That folks understand the almost conversational aspect of SQL vs. that of the dataframe api but the other points make no difference.

If you’re a competent dev/data person and are productive with the dataframe then yay. Also setup and creating test data and such it’s all objects and functions after all — if anything it’s better than the horribad experience of ORMs.

Re: Polars Cloud and Distributed Polars now available

#45

Post by gigatexal »

drej wrote: post #28
robertkoss wrote: post #24

Earlier quoted context omitted.

That is a false dichotomy. You can use SQL tools but still have to choose the instance type. Especially when considering testability and composability, using a DataFrame API inside regular languages like Python is far superior IMO.

As a user? No, I don't have to choose. What I'm saying is that analysts (who this Polars Cloud targets, just like Coiled or Databricks) shouldn't worry about instance types, shuffling performance, join strategies, JVM versions, cross-AZ pricing etc. In most cases, they should just get a connection string and/or a web UI to run their queries, everything abstracted from them. Sure, Python code is more testable and comp…

Again the issue you’re having is the skill level of the audience you keep bringing up not the tool.

Re: Polars Cloud and Distributed Polars now available

#46

Post by drej »

gigatexal wrote: post #45
drej wrote: post #28

Earlier quoted context omitted.

As a user? No, I don't have to choose. What I'm saying is that analysts (who this Polars Cloud targets, just like Coiled or Databricks) shouldn't worry about instance types, shuffling performance, join strategies, JVM versions, cross-AZ pricing etc. In most cases, they should just get a connection string and/or a web UI to run their queries, everything abstracted from them. Sure, Python code is more testable and comp…

Again the issue you’re having is the skill level of the audience you keep bringing up not the tool.

I find it much more beneficial to lower the barrier for entry (oftentimes without any sacrifices) instead of spending time and money on upskilling everyone, just because I like engineering.

Re: Polars Cloud and Distributed Polars now available

#48

Post by mr_toad »

robertkoss wrote: post #12

Love it! Still don't get why one of the biggest player in the space, Databricks is overinvesting in Spark. For startups, Polars or DuckDB are completely sufficient. Other companies like Palantir already support bring your own compute.

Databricks is targeting large enterprises, who have a variety of users. Having both Python and SQL as first class languages is a selling point.

Re: Polars Cloud and Distributed Polars now available

#49

Post by mr_toad »

sirfz wrote: post #40
lvl155 wrote: post #19

Polars is certainly better than pandas doing things locally. But that is a low bar. I’ve not had great experience using Polars on large enough datasets. I almost always end up using duckdb. If I am using SQL at the end of the day, why bother starting with Polars? With AI these days, it’s ridiculously fast to put together performant SQLs. Heck you can even make your own grammar and be done with it.

SQL is definitely easier and faster to compose than any dataframe syntax but I think pandas syntax (via slicing API) is faster to type and in most cases more intuitive but I still use polars for all df-related tasks in my workflow since it's more structured and composable (although needs more time to construct but that's a cost I'm willing to take when not simply prototyping). When in an ipython session, sql via duck…

> SQL is definitely easier and faster to compose

Sometimes. But sometimes Python is just much easier. For example transposing rows and columns.

Post reply on HN