Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern Pandas (Part 2): Method Chaining

tomaugspurger.github.io

Post reply on HN
Print view View original

31–40 of 72 posts

Re: Modern Pandas (Part 2): Method Chaining

#31

Post by biellls »

account-5 wrote: post #11

I've always found pandas really hard to use or reason about. I eventually get there but I don't like the code. Obviously subjective. I've never used another "data science" language though so I've no experience beyond it.

A lot of people hate on SQL. I used to be one of them but I've come to think that for data transformations it's hard to beat. My current favorite is DuckDB, which is like a SQLite but columnar. It has great performance and it's easy to call it from python and even run SQL on pandas dataframes.

Re: Modern Pandas (Part 2): Method Chaining

#32

Post by biellls »

mjburgess wrote: post #22
zasdffaa wrote: post #21

Earlier quoted context omitted.

My python is rusty but IIRC it allows functional stuff (except for expression-only lambdas, boo). From https://docs.python.org/3/howto/functional.html it's got map/filter/currying and plenty more, what's misting in your view?

pattern matching expressions, syntax for partial application and composition, a typing system which can express structural types, a generalised list comprehension

Pattern matching is supported in 3.10 and can match classes structurally, as well as other types (edit: I see that you mean you don't like that it's a statement, which I agree with). The typing system supports structural types with Protocol. Personally what I miss most are multi-line lambdas.

Re: Modern Pandas (Part 2): Method Chaining

#33

Post by otsaloma »

account-5 wrote: post #11

I've always found pandas really hard to use or reason about. I eventually get there but I don't like the code. Obviously subjective. I've never used another "data science" language though so I've no experience beyond it.

Here's another alternative. I wrote Dataiter specifically as I too was frustrated with Pandas. In my experience if you design a new API from scratch (and don't try to reimplement the Pandas API as many projects have done!) and have some vision and consistent principles, it's well possible to get a good intuitive API as a result. Two relevant issues remain: You're limited by NumPy's datatypes and their problems, such as memory-hogging strings and a lack of a proper missing value (NA), and secondly, limited by the Python language, so compared to e.g. dplyr's non-standard evaluation, you'll need to use lambda functions, which are unfortunately clumsy and verbose.

https://github.com/otsaloma/dataiter

Here's a comparison of dplyr vs. Dataiter vs. Pandas, which should give quick overview of the similarieties and differences.

https://dataiter.readthedocs.io/en/latest/_static/comparison...

Re: Modern Pandas (Part 2): Method Chaining

#34

Post by otsaloma »

The examples presented are probably carefully selected, my experience is that if you actually use Pandas with method chaining, (1) you get ugly-looking code due to a mixture of method calls and various different kinds of bracket indexing and (2) you eventually run into things that just can't be (nicely) chained and then you need to break the chain – even new stuff, such as DataFrame.append being deprecated in favor of pd.concat.

Re: Modern Pandas (Part 2): Method Chaining

#35

Post by eesmith »

sterlinm wrote: post #26

This is a great series of articles but a bit funny to call it modern pandas these days since it’s six years old. Has it been updated?

Modern art ended in the 1970's. https://en.wikipedia.org/wiki/Modern_art

Perhaps the successor should be contemporary Pandas, or postmodern Pandas. :)

Re: Modern Pandas (Part 2): Method Chaining

#36

Post by patrick451 »

mayankkaizen wrote: post #27

Pandas is something that I wish I could avoid at any cost but I can't. There is simply no design philosophy. API is as ugly as it gets. I find it greatly unintuitive. It feels like a giant missmash of hacks on top of other hacks. Sometime I wish designers of Numpy or scikit-learn should have developed Pandas.

Numpy has plenty of API warts on its own. It's not obvious to me they would have done a better job at all.

Re: Modern Pandas (Part 2): Method Chaining

#37

Post by jacobdi »

My team has been trying to modernize pandas from a different tact. Regardless of struggle with the syntax, it seems Pandas is very sticky, and we don't predict much migration to other data science languages. Instead of refining the syntax, we have combined it with a spreadsheet GUI (https://github.com/mito-ds/monorepo). Here, we worry less about writing perfect syntax ourselves, and let the GUI write the code for functions like pivot tables and merges that work well visually.

Re: Modern Pandas (Part 2): Method Chaining

#40

Post by smartmic »

account-5 wrote: post #11

I've always found pandas really hard to use or reason about. I eventually get there but I don't like the code. Obviously subjective. I've never used another "data science" language though so I've no experience beyond it.

If you are ready to stretch your mind and have time and energy to learn a new skill, I recommend this as "data science" language: APL

Background: I first gained some experience with J, where I first learned to appreciate the advantages of array languages. The main advantage is actually having your own way of thinking about processing multidimensional data. Now, recently, I've gotten into the notation of APL, and it's really even cooler than the ASCII "noise" of J. The symbols make it easier for me to both write and read programs. Admittedly, for more complex operations with data it takes a lot of learning that you usually don't have. But for simple transformation APL is already quite fast usable and convinces beyond the "mainstream".

Post reply on HN