I've plugged this several times here (just a fan) but chdb's DataStore is a "lazy" drop-in replacement for pandas dataframe. Pandas API with chdb performance. I don't use it myself (I do everything mostly in sql with either duckdb or chdb) but always found pandas more convinient than polars for quick and dirty data crunching (less typing and frankly more pythonic if you're used to slicing).
Pandas Should Go Extinct
51–60 of 105 posts
Re: Pandas Should Go Extinct
#52Cute title; I thought I was about to read a contrarian ecologist saying "a species that is so very specialized in its diet and finicky in its reproductive behavior doesn't deserve, evolutionarily speaking, to survive". (Seriously, it's seriously freakin' difficult to get pandas to reproduce in captivity). And while my knee-jerk reaction would be "maybe, but we should still preserve them because we can", I was prepared to see if the author had a serious argument to present.
Instead it's a cute bait-and-switch title, and the article tells you upfront that it's actually about the Python `pandas` library. Which I think I've encountered maybe once in my entire career (I'm not in the data-science field), so I don't have much meaningful to say about the article itself. I just want to commend the author on fooling me with the title. This is the kind of "clickbait" I can respect and actually wish there was a little bit more of sometimes. A nice chuckle, then a real article.
Re: Pandas Should Go Extinct
#53I'd drop pandas if polars worked eamlessly with sklearn.
Most of the time it works, but there are still corner cases throughout the python ML ecosystem where polars fails.
This and the decision not to natively plot with matplotlib keep me in pandas for most tasks. (Plus there's is still a relatively large demand for pandas training.)
Re: Pandas Should Go Extinct
#54People are usually surprised to hear that polars can be slower for fairly pedestrian operations, especially with smaller datasets. For example take a 5000 x 3 dataframe of float64 and sort by one column and you'll find polars takes about 2.5x as long. If you set POLARS_MAX_THREADS to 1 then it's faster. Though this all depends on the machine. Polars tends to shine with larger datasets or where it can heavily take advantage of query planning.
Don't skip your profiling
Re: Pandas Should Go Extinct
#55Strongly disagree with the author. For dumb simple select group by OLAP queries on medium data ? Sure clickhouse local or duckdb works perfectly well. But if you need to construct dataset ? Or process existing dataset, do heavy filtering, transformation, reshaping, splitting? The proper ETL work, then pandas is really the perfect use case. And pandas can work with small memory footprint as well, its actually trivial…
Use pandas if you want advanced analytics, visualization, or ml. Use SQL if you need to move data around.
How would you use sql to move data from three excel files (some of them may have arbitrary number of worksheets), pack of json files, two csv files, one mysql, one postgres db, some parquets, some of them in S3) ???
And in the end save processed data in aws s3 in another format like iceberg or whatever
In pandas a lot of these are one-liners that are impossible in sql (depends on what sql engine you have?)
Re: Pandas Should Go Extinct
#56Polars seems nice but in my experience using it, the "lazy" APIs would still immediately materialize a ton of stuff in memory and had very spotty support on what data formats and storage integrations were possible with scan_* functions (though that was half a year ago and the support is slowly improving). It's frustrating, I mean really frustrating, to think I could solve a lot of my "scan through heinous amounts of…
Did you ever report these issues to polars developers?
I reported many bugs to both pandas and polars over the years and both teams tend to address relatively big ones. (Still have some outstanding pandas bugs that I think are a big deal but the devs disagree.)
Re: Pandas Should Go Extinct
#57Nitpick for the author: it looks like you've got (a,b] when you actually mean [a,b). Either that or change the ≥ to be >.
Sorry, where are you seeing this? In one of the code samples?
The tables in the https://eddie.codes/posts/pandas-should-go-extinct/#i-have-b... section.
Re: Pandas Should Go Extinct
#58Nitpick for the author: it looks like you've got (a,b] when you actually mean [a,b). Either that or change the ≥ to be >.
If you mean the tables in the https://eddie.codes/posts/pandas-should-go-extinct/#i-have-b... section, it looks like those came straight from the Amazon paper, in which case __eddie__ shouldn't fix them, on general "quote accurately even if the quoted document has mistakes in it" principles. (Though a [sic] might help).
Re: Pandas Should Go Extinct
#59Earlier quoted context omitted.
If you learn to write pandas correctly you end up writing it very similar to polars (or tidyverse). I agree that the API has warts, though typically it is more concise than polars.
The problem is a small change in the question can force you to make huge changes in the code in pandas. I recently gave some examples in my blog [1]. E.g. compare the last two code blocks, where the small change is just that the median is taken within countries. This requires several line changes in pandas. [1] https://bjarkehautop.github.io/Website/blog/data-wrangling-t...
Personally, I think the inclusion of lambda functions in the pandas code shows that the approach isnt ideal to begin with. Fine on smaller datasets, but typically becomes a slow and cumbersome overhead when interacting with larger datasets. More verbose per line vector operations are generally orders of magnitude faster, and become more so as the data grows. They also tend to be easier to read / identify what is going on or what the intent is for people unfamiliar with the code base.
Re: Pandas Should Go Extinct
#60Earlier quoted context omitted.
If you learn to write pandas correctly you end up writing it very similar to polars (or tidyverse). I agree that the API has warts, though typically it is more concise than polars.
The problem is a small change in the question can force you to make huge changes in the code in pandas. I recently gave some examples in my blog [1]. E.g. compare the last two code blocks, where the small change is just that the median is taken within countries. This requires several line changes in pandas. [1] https://bjarkehautop.github.io/Website/blog/data-wrangling-t...
I looked at your code very quickly, but it looks like you need to use .filter after a .groupby...