Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas Should Go Extinct

eddie.codes

Post reply on HN
Print view View original

91–100 of 105 posts

Re: Pandas Should Go Extinct

#91

Post by fithisux »

The article is harsh for no reason. Pandas served us, for a lot of time and was the de-facto library for Data Scientists. It evolves. Polars was a rewrite following bets practices learnt while working with Pandas.

I do not understand the aggressiveness towards previous evolutionary steps. Who knows, may they come up with newer methods. I still use it. I also use DuckDB and Polars. DuckDB is more useful to me these days from either. Still Pandas has a place in my arsenal. Mostly because I know it better and I like it.

There is also libraries for doing type checking.

Re: Pandas Should Go Extinct

#95

Post by __mharrison__ »

anakaine wrote: post #68
__mharrison__ wrote: post #61

Earlier quoted context omitted.

This is a common complaint I get all the time (heard it this week while teaching pandas). I compare this to whitespace indentation in Python. Lots of folks complain about it before using it. After they use it it is a non issue. If it really is an issue (and it generally isn't a cause of vectorization removal when used correctly) and you can't get over the syntactic noise if the lambda, pandas 3 introduced pd.col (tha…

Ive seen it in such a wide variety of scenarios where it causes vectorisation removal because of its implementation that I dont reach for it by default any more. I've a number of devs around me who have similar opinions. Each know how to use it, but you dont always write the code you're reviewing or optimising.

Would love to see examples.

Re: Pandas Should Go Extinct

#96

Post by __mharrison__ »

latent-person wrote: post #62
__mharrison__ wrote: post #60

Earlier quoted context omitted.

I looked at your code very quickly, but it looks like you need to use .filter after a .groupby...

The text right above the code says why you can't... edit: Let me clarify. From the blog-post: > since a `DataFrameGroupBy` object doesn’t have a `.query()` or boolean-indexing shortcut of its own, so filtering within groups needs `.apply()` again, and the surrounding pipeline has to be rebuilt around it: Hence you really do need one of the versions of the code I gave. You can't do the naive approach with just `.group…

I misspoke, you need to use .groupby/.transform to add a new filtering column:

    (sales
      .assign(country_median=lambda df_: (
          df_.groupby("country")["amount"].transform("median")
      ))
      .query("amount

Re: Pandas Should Go Extinct

#97

Post by jamincan »

__mharrison__ wrote: post #44

I'm in the middle of wrapping up the edits for Effective Pandas 3rd Edition. (I also wrote a Polars book and just wrapped up a weeklong training session on pandas this week.) Pandas is not perfect, it has a bunch of warts. But it is good enough for most. (And many of those folks are using Excel or tableau or power bi... These were the types I was training this week). If you have medium data, migrating from pyarrow ba…

Out of curiousity, why wouldn't someone just migrate directly to Polars?

Re: Pandas Should Go Extinct

#98

Post by gcr »

minimaxir wrote: post #4

It's been a while since I've seen an actual data science post submitted to Hacker News: both because AI has superset a lot of DS tasks (e.g. vector embeddings), but also because not much new has happened in DS. Polars has been around for a bit and as noted it is much better than pandas, but otherwise the DS ecosystem has been somewhat stagnant. I'd write more tutorials about how to use data science tooling but one co…

What do you mean? Data science is advancing at an incredible pace IMO! DuckDB is the coolest thing to happen to data science in my neck of the woods

Re: Pandas Should Go Extinct

#99

Post by latent-person »

__mharrison__ wrote: post #96
latent-person wrote: post #62

Earlier quoted context omitted.

The text right above the code says why you can't... edit: Let me clarify. From the blog-post: > since a `DataFrameGroupBy` object doesn’t have a `.query()` or boolean-indexing shortcut of its own, so filtering within groups needs `.apply()` again, and the surrounding pipeline has to be rebuilt around it: Hence you really do need one of the versions of the code I gave. You can't do the naive approach with just `.group…

I misspoke, you need to use .groupby/.transform to add a new filtering column: (sales .assign(country_median=lambda df_: ( df_.groupby("country")["amount"].transform("median") )) .query("amount

Yes, so basically equivalent to the code I showed in the blog.

Re: Pandas Should Go Extinct

#100

Post by Grimblewald »

parineum wrote: post #78
Grimblewald wrote: post #75

Earlier quoted context omitted.

because breeding intelligent creatures generally requires keeping them happy. When treated like a farm aniaml, many high cognition anaimals will refuse to reproduce, just look at how bad life has become for average humans, and predictably, humans are starting to refuse to reproduce.

Birth rates fall as prosperity increases.

to replacement, we're rapidly falling below. distributuon and related trends are extremely grim.
Post reply on HN