Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python Data Science Handbook

jakevdp.github.io

Post reply on HN
Print view View original

41–50 of 67 posts

Re: Python Data Science Handbook

#41

Post by crystal_revenge »

xenophonf wrote: post #6
sschnei8 wrote: post #4

Interesting choice of Pandas in this day and age. Maybe he’s after imparting general concepts that you could apply to any tabular data manipulator rather than selecting for the latest shiny tool.

What's wrong with Pandas?

Pandas is generally awful unless you're just living in a notebook (and even then it's probably least favorite implementation of the 'data frame' concept).

Since Pandas lacks Polars' concept of an Expression, it's actually quite challenging to programmatically interact with non-trivial Pandas queries. In Polars the query logic can be entirely independent of the data frame while still referencing specific columns of the data frame. This makes Polars data frames work much more naturally with typical programming abstractions.

Pandas multi-index is a bad idea in nearly all contexts other than it's original use case: financial time series (and I'll admit, if you're working with purely financial time series, then Pandas feels much better). Sufficiently large Pandas code bases are littered with seemingly arbitrary uses of 'reset_index', there are many times where multi-index will create bugs, and, most important, I've never seen any non-financial scenario where anyone has ever used Multi-index to their advantage.

Finally Pandas is slow, which is honestly the least priority for me personally, but using Polars is so refreshing.

What other data frames have you used? Having used R's native dataframes extensively (the way they make use of indexing is so much nicer) in addition to Polars both are drastically preferable to Pandas. My experience is that most people use Pandas because it has been the only data frame implementation in Python. But personally I'd rather just not use data frames if I'm forced to used Pandas. Could you expand on what you like about Pandas over other data frames models you've worked with?

Re: Python Data Science Handbook

#42

Post by crystal_revenge »

biofox wrote: post #17
clickety_clack wrote: post #8

Earlier quoted context omitted.

I probably wouldn’t rewrite an entire data science stack that used pandas, but most people would use polars if starting a new project today.

R and Matlab workflows have been fairly stable for the past decade. Why is the Python ecosystem so... unstable? It puts me off investing any time in it.

I love R, but how can you make that claim when R uses three distinct object-oriented systems all at the same time? R might seem stable only because it carries along with it 50 years of history of programming languages (part of it's charm, where else can you see the generic function approach to OOP in a language that's still evolving?)

Finally, as someone who wrote a lot of R pre-tidyverse, I've seen the entire ecosystem radically change over my career.

Re: Python Data Science Handbook

#43

Post by pid-1 »

Pandas is cancer. Please stop teaching it to people.

Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed).

Pandas code is untestable, unreadable, hard to refactor and impossible to reuse.

Trillions of dollars are wasted every year by people having to rewrite pandas code.

Re: Python Data Science Handbook

#44

Post by refactor_master »

I honestly don't get why you'd hate pandas more than anything else in the Python ecosystem. It's probably not the best tool in the world, and sure, like everybody else I'd rewrite the universe in Rust if I could start over, and had infinite time to catch up.

But the code base I work on has thousands and THOUSANDS of lines of Pandas churning through big data, and I can't remember the last time it lead to a bug or error in production.

We use pandas + static schema wrapper + type checker, so you'll have to get exotic to break things.

Re: Python Data Science Handbook

#45

Post by mttpgn »

pid-1 wrote: post #43

Pandas is cancer. Please stop teaching it to people. Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed). Pandas code is untestable, unreadable, hard to refactor and impossible to reuse. Trillions of dollars are wasted every year by people having to rewrite pandas code.

> Pandas code is untestable

The thousand-plus data integrity tests I've written in pandas tell a different story...

Re: Python Data Science Handbook

#46

Post by physicsguy »

pid-1 wrote: post #43

Pandas is cancer. Please stop teaching it to people. Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed). Pandas code is untestable, unreadable, hard to refactor and impossible to reuse. Trillions of dollars are wasted every year by people having to rewrite pandas code.

I found Pandera quite good for wrapping input/output expectations over Pandas. At the end of the day the vectorisation of operations in it and other table based formats mean they’re not easy to replace performantly.

Re: Python Data Science Handbook

#47

Post by mulmboy »

pid-1 wrote: post #43

Pandas is cancer. Please stop teaching it to people. Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed). Pandas code is untestable, unreadable, hard to refactor and impossible to reuse. Trillions of dollars are wasted every year by people having to rewrite pandas code.

> Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed).

I see this take somewhat often, and usually with similar lack of nuance. How do you come to this? In other cases where I've seen this it's from people who haven't worked in any context where performance or scientific computing ecosystem interoperability matters - missing a massive part of the picture. I've struggled to get through to them before. Genuine question.

Re: Python Data Science Handbook

#48

Post by isolatedsystem »

pid-1 wrote: post #43

Pandas is cancer. Please stop teaching it to people. Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed). Pandas code is untestable, unreadable, hard to refactor and impossible to reuse. Trillions of dollars are wasted every year by people having to rewrite pandas code.

I've recently had to migrate over to Python from Matlab. Pandas has been doing my head in. The syntax is so unintuitive. In Matlab, everything begins with a `for` loop. Inelegant and slow, yes, but easy to reason about. Easy to see the scope and domain of the problem, to visualise the data wrangling.

Pandas insist you never use a for loop. So, I feel guilty if I ever need a throwaway variable on the way to creating a new column. Sometimes methods are attached to objects, other times they aren't. And if you need to use a function that isn't vectorised, you've got to do df.apply anyway. You have to remember to change the 'axis' too. Plotting is another thing that I can't get my head around. Am I supposed to use Pandas' helpers like df.plot() all the time? Or ditch it and use the low level matplotlib directly? What is idiomatic? I cannot find answers to much of it, even with ChatGPT. Worse, I can't seem to create a mental model of what Pandas expects me to do in a given situation.

Pandas has disabused me of the notion that Python syntax is self-explanatory and executable-pseudocode. I find it terrible to look at. Matlab was infinitely more enjoyable.

Re: Python Data Science Handbook

#50

Post by radus »

isolatedsystem wrote: post #48
pid-1 wrote: post #43

Pandas is cancer. Please stop teaching it to people. Everything it does can be done reasonable well with list comprehensions and objects that support type annotations and runtime type checking (if needed). Pandas code is untestable, unreadable, hard to refactor and impossible to reuse. Trillions of dollars are wasted every year by people having to rewrite pandas code.

I've recently had to migrate over to Python from Matlab. Pandas has been doing my head in. The syntax is so unintuitive. In Matlab, everything begins with a `for` loop. Inelegant and slow, yes, but easy to reason about. Easy to see the scope and domain of the problem, to visualise the data wrangling. Pandas insist you never use a for loop. So, I feel guilty if I ever need a throwaway variable on the way to creating a…

Polars has a much more consistent API, give it a shot.

Regarding your plotting question: use seaborn when you can, but you’ll still need to know matplotlib.

Post reply on HN