61–67 of 67 posts
I honestly don't get why you'd hate pandas more than anything else in the Python ecosystem. It's probably not the best tool in the world, and sure, like everybody else I'd rewrite the universe in Rust if I could start over, and had infinite time to catch up. But the code base I work on has thousands and THOUSANDS of lines of Pandas churning through big data, and I can't remember the last time it lead to a bug or erro…
Pandas is cancer. Please stop teaching it to people. Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed). Pandas code is untestable, unreadable, hard to refactor and impossible to reuse. Trillions of dollars are wasted every year by people having to rewrite pandas code.
I've recently had to migrate over to Python from Matlab. Pandas has been doing my head in. The syntax is so unintuitive. In Matlab, everything begins with a `for` loop. Inelegant and slow, yes, but easy to reason about. Easy to see the scope and domain of the problem, to visualise the data wrangling. Pandas insist you never use a for loop. So, I feel guilty if I ever need a throwaway variable on the way to creating a…
A for loop is a lot about the "how" but apply, join etc are much closer to the "what".
Earlier quoted context omitted.
Code using pandas is testable and reusable in much the same way as any other code, make functions that take and return data. That said, the polars/narwals style API is better than pandas API for sure. More readable and composable, simpler (no index) and a bit less weird overall.
Polars made the mistake of not maintaining row order for all operations, via the False-by-default argument of maintain_order. This is basically the billion-dollar null mistake for data frames.
Earlier quoted context omitted.
What's wrong with Pandas?
Pandas is generally awful unless you're just living in a notebook (and even then it's probably least favorite implementation of the 'data frame' concept). Since Pandas lacks Polars' concept of an Expression, it's actually quite challenging to programmatically interact with non-trivial Pandas queries. In Polars the query logic can be entirely independent of the data frame while still referencing specific columns of th…
I like how in Pandas (and in R), I can quickly load data sets up in a manner that lets me do relational queries using familiar syntax. For my Elite: Dangerous project, because I couldn't get Pandas to work for me (which the reader should chalk up to my ignorance and not any deficiency of Pandas itself), I ended up using the SQLAlchemy ORM with Marshmallow to load the data into SQLite or PostgreSQL. Looking back at the work, I probably ought to have thrown it into a JSON-aware data warehouse somehow, which I think is how the guy behind Spansh does it, but I'm not a big data guy (yet) and have a lot to learn about what's possible.
I honestly don't get why you'd hate pandas more than anything else in the Python ecosystem. It's probably not the best tool in the world, and sure, like everybody else I'd rewrite the universe in Rust if I could start over, and had infinite time to catch up. But the code base I work on has thousands and THOUSANDS of lines of Pandas churning through big data, and I can't remember the last time it lead to a bug or erro…
Custom schema wrapper or some package you'd recommend from pypi?
* Mypy dependency and really bad PEP compliance * Sub-optimal runtime check decorators * Subclasses pd.DataFrame, so using e.g. .assign(...) makes the type checker think it's still the same type, but now you just violated your own schema
So I wrote my own library that solves all these issues, but it's currently company-internal. I've been meaning to push for open-sourcing it, but just haven't had the time.
This book was absolute fire for getting started with data science in 2017-2018, Jake is a great teacher.
it's written 8 years ago though, there is a 2ed of the book by the same author.