Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas Should Go Extinct

eddie.codes

Post reply on HN
Print view View original

31–40 of 105 posts

Re: Pandas Should Go Extinct

#32

Post by __eddie__ »

akdor1154 wrote: post #30

Sup Eddie, the actual motivating example is we finally nixed Pandas from Data Ingestion, now reading excel files takes 2 seconds instead of 2 minutes. However unfortunately your > TODO: rewrite this entire service remains.

> Sup Eddie, the actual motivating example is we finally nixed Pandas from Data Ingestion, now reading excel files takes 2 seconds instead of 2 minutes. However unfortunately your

I shall pass this information along to my sleep paralysis demons. They'll be glad to hear it.

> > TODO: rewrite this entire service

Part of me enjoys the historical significance of this. However, I feel like keeping it would violate my own princples: https://eddie.codes/posts/source-code-comments/

Re: Pandas Should Go Extinct

#33

Post by bijowo1676 »

Strongly disagree with the author.

For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well.

But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case.

And pandas can work with small memory footprint as well, its actually trivial to do that, plus there are libraries like Modin that are upgrades over pandas with pandas api

Pandas is the swiss knife tool of data science that lets you do anything with the data and it integrates well with ML libraries

Re: Pandas Should Go Extinct

#35

Post by minimaxir »

bijowo1676 wrote: post #33

Strongly disagree with the author. For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well. But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case. And pandas can work with small memory footprint as well, its actually trivial…

It's subjective, but IMO polars's APIs are better and than pandas's for ETL. Notably, pandas ported over polars's use of .col() for the 3.0.0 release: https://pandas.pydata.org/docs/whatsnew/v3.0.0.html#initial-...

Also, pandas's group_by() and window function pipelines made me want to tear my hair out while both are easy in polars.

Re: Pandas Should Go Extinct

#37

Post by minimaxir »

jgalt212 wrote: post #23

I'd drop pandas if polars worked eamlessly with sklearn.

You can do to_pandas() and from_pandas() which doesn't have much overhead, or to_numpy() directly which is typically zero-copy and thus no overhead: https://docs.pola.rs/api/python/stable/reference/dataframe/a...

Re: Pandas Should Go Extinct

#38

Post by dweinus »

Nice post but they quickly disregard Dask, don't explain why, don't test it, even exclude it from the benchmark they quote. I don't know if is the better answer, but it seems worth testing if you want a balance approachable + scalable. That's kind the thing Dask was meant to do.

Re: Pandas Should Go Extinct

#39

Post by latent-person »

In my opinion a better argument to stop using pandas is the very unintuitive API pandas have. Additionally, a slight change in the query can force you to restructure the whole query (change all lines), while in Polars (and tidyverse in R) it's just a simple one-line change.
Post reply on HN