Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas 1.0

pandas.pydata.org

Post reply on HN
Print view View original

21–30 of 80 posts

Re: Pandas 1.0

#21

Post by spectramax »

_coveredInBees wrote: post #19
closed wrote: post #14

I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…

I think another area pandas has done a lot of work on is with datetimes. Numpy's datetime objects are pretty deficient when you need to perform computations / data wrangling with them and utilizing python's native datetime objects would slow things down a decent bit. So they have done a lot of work to create their own datetime implementation that helps a lot when dealing with tabular data and performing date/time bas…

Datetimes are still a mess. Now we have 3 datetime objects and converting between them is not obvious or trivial:

https://i.stack.imgur.com/uiXQd.png

Re: Pandas 1.0

#22

Post by prepend »

closed wrote: post #14

I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…

I like that pandas 1.0 took “Int64” out of trial mode.

It wasn’t functionally very different from everything being a float, but I will appreciate not having to format floats as ints in all my reports.

Re: Pandas 1.0

#23

Post by closed »

prepend wrote: post #22
closed wrote: post #14

I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…

I like that pandas 1.0 took “Int64” out of trial mode. It wasn’t functionally very different from everything being a float, but I will appreciate not having to format floats as ints in all my reports.

Ah, same! I'm also relieved to hopefully never again have to say: "whoops I converted nan into a very large integer".

Re: Pandas 1.0

#24

Post by _coveredInBees »

spectramax wrote: post #21
_coveredInBees wrote: post #19

Earlier quoted context omitted.

I think another area pandas has done a lot of work on is with datetimes. Numpy's datetime objects are pretty deficient when you need to perform computations / data wrangling with them and utilizing python's native datetime objects would slow things down a decent bit. So they have done a lot of work to create their own datetime implementation that helps a lot when dealing with tabular data and performing date/time bas…

Datetimes are still a mess. Now we have 3 datetime objects and converting between them is not obvious or trivial: https://i.stack.imgur.com/uiXQd.png

Yeah sure, but that isn't their fault really. I can see a role for python's datetime being separate from numpy/pandas, but I do think a consolidated datetime object would be better to have rather than the numpy and pandas versions that are similar but not the same.

Re: Pandas 1.0

#25

Post by Aperocky »

_coveredInBees wrote: post #7

Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…

> yet the API is all over the place

This so much. I've answered quite a bit of pandas on SO, and I have to say the APIs are a mess. There are always multiple ways to do things, there are hidden traps that can lead to huge run times, and stuff that are just wildly un-pythonic.

It's still the best general data processing has to offer. But a smaller, cleaner package might just take the cake.

Re: Pandas 1.0

#26

Post by spectramax »

_coveredInBees wrote: post #24
spectramax wrote: post #21

Earlier quoted context omitted.

Datetimes are still a mess. Now we have 3 datetime objects and converting between them is not obvious or trivial: https://i.stack.imgur.com/uiXQd.png

Yeah sure, but that isn't their fault really. I can see a role for python's datetime being separate from numpy/pandas, but I do think a consolidated datetime object would be better to have rather than the numpy and pandas versions that are similar but not the same.

Definitely agree, pandas/numpy can have internal representations of efficient datetimes but the user should not have to deal with these conversions.

Re: Pandas 1.0

#27

Post by spectramax »

Aperocky wrote: post #25
_coveredInBees wrote: post #7

Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…

> yet the API is all over the place This so much. I've answered quite a bit of pandas on SO, and I have to say the APIs are a mess. There are always multiple ways to do things, there are hidden traps that can lead to huge run times, and stuff that are just wildly un-pythonic. It's still the best general data processing has to offer. But a smaller, cleaner package might just take the cake.

I can see a future for pytonic interface to pandas. It would run pandas in the backend but would be just as easy as manipulating list, dicts, sets and tuples...we'd have a 5th type - tables.

Re: Pandas 1.0

#28

Post by mrfusion »

Assuming you want to use python. What’s the alternative to pandas? Kind of a brew up your own code kind of thing? Csv module I guess?

Re: Pandas 1.0

#30

Post by 322849a8d87 »

_coveredInBees wrote: post #7

Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…

> yet the API is all over the place

Agreed. In particular one might have hoped that 1.0 would fix indexing. .ix (deprecated), .loc, .iloc and "[" is an example of what people mean by saying the API is (a) a mess and (b) "deeply unpythonic". Shouldn't "[" be removed entirely if .loc and .iloc are recommended, given the odd and unpredictable edge cases with "["?

> unless you use it regularly for a long period of time, it is almost impossible to get fluent with it.

Agreed.

I know a huge amount of valiant, voluntary, open source work has gone into it, but it is a shame that the primary data-frame library in the Python ecosystem lacks a clean, pythonic API. Having been negative I don't want to obscure the fact that it does have some great and powerful code behind its API.

Post reply on HN