Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Modern Pandas (Part 2): Method Chaining

tomaugspurger.github.io

Post reply on HN
Print view View original

21–30 of 72 posts

Re: Modern Pandas (Part 2): Method Chaining

#21

Post by zasdffaa »

mjburgess wrote: post #12
account-5 wrote: post #11

I've always found pandas really hard to use or reason about. I eventually get there but I don't like the code. Obviously subjective. I've never used another "data science" language though so I've no experience beyond it.

Everyone does. Perhaps if python would add more support for FP (rather than its present hostility), we'd be able to phrase data transformations more naturally in the language.

My python is rusty but IIRC it allows functional stuff (except for expression-only lambdas, boo).

From https://docs.python.org/3/howto/functional.html it's got map/filter/currying and plenty more, what's misting in your view?

Re: Modern Pandas (Part 2): Method Chaining

#22

Post by mjburgess »

zasdffaa wrote: post #21
mjburgess wrote: post #12

Earlier quoted context omitted.

Everyone does. Perhaps if python would add more support for FP (rather than its present hostility), we'd be able to phrase data transformations more naturally in the language.

My python is rusty but IIRC it allows functional stuff (except for expression-only lambdas, boo). From https://docs.python.org/3/howto/functional.html it's got map/filter/currying and plenty more, what's misting in your view?

pattern matching expressions, syntax for partial application and composition, a typing system which can express structural types, a generalised list comprehension

Re: Modern Pandas (Part 2): Method Chaining

#23

Post by zasdffaa »

mjburgess wrote: post #22
zasdffaa wrote: post #21

Earlier quoted context omitted.

My python is rusty but IIRC it allows functional stuff (except for expression-only lambdas, boo). From https://docs.python.org/3/howto/functional.html it's got map/filter/currying and plenty more, what's misting in your view?

pattern matching expressions, syntax for partial application and composition, a typing system which can express structural types, a generalised list comprehension

> pattern matching expressions,

nothing intrinsically 'functional' about this, though it's nice

> syntax for partial application

https://docs.python.org/3/library/functools.html#functools.p...>

> and composition,

Hmm, to my surprise I can't find this but TBH it's trivial to write, here's an eg https://stackoverflow.com/questions/16739290/composing-funct...>

> a typing system which can express structural types,

typing is orthogonal to FP (though very nice to have)

> a generalised list comprehension

IDK what this means, what's ungeneral about comprehensions currently?

Re: Modern Pandas (Part 2): Method Chaining

#25

Post by nojito »

riyadparvez wrote: post #19
tomtom1337 wrote: post #14

Earlier quoted context omitted.

I'd recommend checking out polars as an alternative to pandas - https://github.com/pola-rs/polars It has a rather different api, and is significantly faster. Highly recommend it.

The issue is there's a huge ecosystem developed on top of Pandas. So if it's different API, now you lose access to all the ecosystem.

Polars can export to arrow which you can then load into pandas.

Or you can convert it directly into a pandas Df

Re: Modern Pandas (Part 2): Method Chaining

#27

Post by mayankkaizen »

Pandas is something that I wish I could avoid at any cost but I can't. There is simply no design philosophy. API is as ugly as it gets. I find it greatly unintuitive. It feels like a giant missmash of hacks on top of other hacks.

Sometime I wish designers of Numpy or scikit-learn should have developed Pandas.

Re: Modern Pandas (Part 2): Method Chaining

#28

Post by ritchie46 »

riyadparvez wrote: post #19
tomtom1337 wrote: post #14

Earlier quoted context omitted.

I'd recommend checking out polars as an alternative to pandas - https://github.com/pola-rs/polars It has a rather different api, and is significantly faster. Highly recommend it.

The issue is there's a huge ecosystem developed on top of Pandas. So if it's different API, now you lose access to all the ecosystem.

Should not be a real blocker. There is good interop with arrow, numpy and pandas. Where arrow and numpy mostly is zero-copy.

So you are one `df.to_numpy()/df.to_pandas()` away to `X` libary you want to use.

Re: Modern Pandas (Part 2): Method Chaining

#29

Post by mjburgess »

zasdffaa wrote: post #23
mjburgess wrote: post #22

Earlier quoted context omitted.

pattern matching expressions, syntax for partial application and composition, a typing system which can express structural types, a generalised list comprehension

> pattern matching expressions, nothing intrinsically 'functional' about this, though it's nice > syntax for partial application https://docs.python.org/3/library/functools.html#functools.p... > > and composition, Hmm, to my surprise I can't find this but TBH it's trivial to write, here's an eg https://stackoverflow.com/questions/16739290/composing-funct... > > a typing system which can express structural types, typi…

expressing generic data transformations naturally in syntax, requires: being able to wrap/unwrap data types (ie., pattern matching), composing operations, partially applying them, and providing syntax to support filter/map/flatMap on arbitrary data structures

Pythons failure to provide this, indeed, outright hostility to doing so is half the reason pandas is a mess of incomprehensible syntax

By prioritising assignment expressions and implementing pattern "matching" as a statement, they're clearly showing a lot of hostility to one of the major use cases of their language

(incidentally, recall that C# introduced LINQ in 2008, a generalised comprehension! That's C#.)

There isnt the syntax to reimplement to support natural phrasings of data transformation, in lieu of this, pandas exploits weird operators such as `.loc[a,b]`.

At some point the dam is going to have to break, and python is going to have to introduce something to resolve this mess. However, i'd bet it'll be a decade of tooth-and-nail fighting about it. It isnt a software engineering language for education any more, and i'm not seeing this reality being acknowledged

Post reply on HN