Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas Should Go Extinct

eddie.codes

Post reply on HN
Print view View original

61–70 of 105 posts

Re: Pandas Should Go Extinct

#61

Post by __mharrison__ »

anakaine wrote: post #59
latent-person wrote: post #49

Earlier quoted context omitted.

The problem is a small change in the question can force you to make huge changes in the code in pandas. I recently gave some examples in my blog [1]. E.g. compare the last two code blocks, where the small change is just that the median is taken within countries. This requires several line changes in pandas. [1] https://bjarkehautop.github.io/Website/blog/data-wrangling-t...

Personally, I think the inclusion of lambda functions in the pandas code shows that the approach isnt ideal to begin with. Fine on smaller datasets, but typically becomes a slow and cumbersome overhead when interacting with larger datasets. More verbose per line vector operations are generally orders of magnitude faster, and become more so as the data grows. They also tend to be easier to read / identify what is goin…

This is a common complaint I get all the time (heard it this week while teaching pandas). I compare this to whitespace indentation in Python.

Lots of folks complain about it before using it. After they use it it is a non issue.

If it really is an issue (and it generally isn't a cause of vectorization removal when used correctly) and you can't get over the syntactic noise if the lambda, pandas 3 introduced pd.col (that work in most (I filed a big about some exceptions) places when you'd use lambda).

Re: Pandas Should Go Extinct

#62

Post by latent-person »

__mharrison__ wrote: post #60
latent-person wrote: post #49

Earlier quoted context omitted.

The problem is a small change in the question can force you to make huge changes in the code in pandas. I recently gave some examples in my blog [1]. E.g. compare the last two code blocks, where the small change is just that the median is taken within countries. This requires several line changes in pandas. [1] https://bjarkehautop.github.io/Website/blog/data-wrangling-t...

I looked at your code very quickly, but it looks like you need to use .filter after a .groupby...

The text right above the code says why you can't...

edit:

Let me clarify. From the blog-post:

> since a `DataFrameGroupBy` object doesn’t have a `.query()` or boolean-indexing shortcut of its own, so filtering within groups needs `.apply()` again, and the surrounding pipeline has to be rebuilt around it:

Hence you really do need one of the versions of the code I gave. You can't do the naive approach with just `.groupby().filter(lambda: )`, since you need a row-wise decision.

Re: Pandas Should Go Extinct

#63

Post by __mharrison__ »

bijowo1676 wrote: post #55
__mharrison__ wrote: post #48

Earlier quoted context omitted.

Use pandas if you want advanced analytics, visualization, or ml. Use SQL if you need to move data around.

How would you use sql to move data from three excel files (some of them may have arbitrary number of worksheets), pack of json files, two csv files, one mysql, one postgres db, some parquets, some of them in S3) ??? And in the end save processed data in aws s3 in another format like iceberg or whatever In pandas a lot of these are one-liners that are impossible in sql (depends on what sql engine you have?)

Those are generally rules of thumbs for data pipelines. (Plus duckdb can read Excel these days I think).

Re: Pandas Should Go Extinct

#64

Post by winwang »

mulmboy wrote: post #54

People are usually surprised to hear that polars can be slower for fairly pedestrian operations, especially with smaller datasets. For example take a 5000 x 3 dataframe of float64 and sort by one column and you'll find polars takes about 2.5x as long. If you set POLARS_MAX_THREADS to 1 then it's faster. Though this all depends on the machine. Polars tends to shine with larger datasets or where it can heavily take adv…

(Without profiling or looking into this at all) I'd guess this has to with thread creation, inter-core communication/latency, and possibly having to merge results or otherwise interleave operations. SMT is another likely candidate.

Regardless, CPUs are really good at single-thread.

Re: Pandas Should Go Extinct

#65

Post by __mharrison__ »

latent-person wrote: post #62
__mharrison__ wrote: post #60

Earlier quoted context omitted.

I looked at your code very quickly, but it looks like you need to use .filter after a .groupby...

The text right above the code says why you can't... edit: Let me clarify. From the blog-post: > since a `DataFrameGroupBy` object doesn’t have a `.query()` or boolean-indexing shortcut of its own, so filtering within groups needs `.apply()` again, and the surrounding pipeline has to be rebuilt around it: Hence you really do need one of the versions of the code I gave. You can't do the naive approach with just `.group…

I'm confused, you can use filter after a groupby in pandas...

It's late here, I'm going to bed, perhaps I'll write the code tomorrow when I'm at my laptop and not on my phone.

Re: Pandas Should Go Extinct

#66

Post by [deleted] »

__mharrison__ wrote: post #65
latent-person wrote: post #62

Earlier quoted context omitted.

The text right above the code says why you can't... edit: Let me clarify. From the blog-post: > since a `DataFrameGroupBy` object doesn’t have a `.query()` or boolean-indexing shortcut of its own, so filtering within groups needs `.apply()` again, and the surrounding pipeline has to be rebuilt around it: Hence you really do need one of the versions of the code I gave. You can't do the naive approach with just `.group…

I'm confused, you can use filter after a groupby in pandas... It's late here, I'm going to bed, perhaps I'll write the code tomorrow when I'm at my laptop and not on my phone.

[deleted]

Re: Pandas Should Go Extinct

#67

Post by MisterMunchkin »

rmunn wrote: post #52

Cute title; I thought I was about to read a contrarian ecologist saying "a species that is so very specialized in its diet and finicky in its reproductive behavior doesn't deserve, evolutionarily speaking, to survive". (Seriously, it's seriously freakin' difficult to get pandas to reproduce in captivity). And while my knee-jerk reaction would be "maybe, but we should still preserve them because we can", I was prepare…

Yeah I wanted the controversial panda article. Can someone write that instead?

Re: Pandas Should Go Extinct

#68

Post by anakaine »

__mharrison__ wrote: post #61
anakaine wrote: post #59

Earlier quoted context omitted.

Personally, I think the inclusion of lambda functions in the pandas code shows that the approach isnt ideal to begin with. Fine on smaller datasets, but typically becomes a slow and cumbersome overhead when interacting with larger datasets. More verbose per line vector operations are generally orders of magnitude faster, and become more so as the data grows. They also tend to be easier to read / identify what is goin…

This is a common complaint I get all the time (heard it this week while teaching pandas). I compare this to whitespace indentation in Python. Lots of folks complain about it before using it. After they use it it is a non issue. If it really is an issue (and it generally isn't a cause of vectorization removal when used correctly) and you can't get over the syntactic noise if the lambda, pandas 3 introduced pd.col (tha…

Ive seen it in such a wide variety of scenarios where it causes vectorisation removal because of its implementation that I dont reach for it by default any more. I've a number of devs around me who have similar opinions. Each know how to use it, but you dont always write the code you're reviewing or optimising.

Re: Pandas Should Go Extinct

#70

Post by Helmut10001 »

Data scientist here. I still use pandas. It is a work horse. But today, the progress in data science is much more in integrating sources and outputs. I find Grist (via API) and n8n much more applicable and process oriented than core DS pandas - although all of these components fit together somewhere. Most of the time, there is very little binding glue needed, e.g. a bash script or standard python file. DS is still fun, but it is much more process oriented today!
Post reply on HN