Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pandas 1.0

pandas.pydata.org

Post reply on HN
Print view View original

61–70 of 80 posts

Re: Pandas 1.0

#61

Post by mirekrusin »

appleiigs wrote: post #59
ImaCake wrote: post #46

Earlier quoted context omitted.

> I have a python/pandas script that take 3 days to run. Pandas .map() and .apply() get real slow on big datasets. I found it quicker to solve a problem with a million line dataset by just using base python iterables instead, so nothing needed to fit into my RAM and I didn't have to work with slow pandas mapping.

Yes, i was thinking about removing the Pandas code and using python iterables. The issue is I did a lot exploration with pandas, which it was good at. If I started with Julia, i wouldn’t need to refactor seeking performance by removing pandas, or for numba, dask, etc. For the existing project i’m thinking the switch to Julia + DataFrames library (despite it being a completely different language) is more of a 1:1 port…

I hear you can use python from Julia, have you tried this as, possibly temporary/transition, approach?

Re: Pandas 1.0

#62

Post by dmos62 »

spectramax wrote: post #47
appleiigs wrote: post #39

Earlier quoted context omitted.

I started using Julia recently. It seems like Julia has been able to take the good parts of Python and iron out the quirks. For example, I'm guessing the Julia DataFrame library is a knock off of Pandas, but the syntax more intuitive and concise - and I can remember it. For Julia itself, the syntax is very similar to Python but doesn't have the weird lambda functions. It has the Javascript style arrow for short anony…

Julia doesn't feel production ready at all. Its fine to mess around in notebooks but I would never recommend it for production use. Not even at a gunpoint. Debugger support is almost non-existent. Using Atom/Juno IDE is a D-grade experience. Julia offers little help to debug problems - errors are almost always without failure - completely tangent to what the real issue is. Julia takes forever to start, syntax was won…

> Not even at a gunpoint

Really? I'd recommend assembler at gunpoint; even at stick-point for that matter. You either spin a mean hyperbole or you're one serious programmer.

I've had the same two complaints as you when I tried Julia ~2 years ago. I was told that the startup situation improved to the point of it not being a problem anymore, but even then people were just reusing Julia's processes. Don't know about the state of errors, but I'd expect noticable improvements there as well, as with most new languages.

I'm anxious to one day come back to Julia, due to its focus on numerical computation, but at the moment it's somewhat counter-balanced by Python ecosystem's maturity.

Re: Pandas 1.0

#63

Post by fifnir »

_coveredInBees wrote: post #7

Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…

pd.read_csv(filepath, sep="\t", header=None)

>>TypeError: Passing a bool to header is invalid.

Fuck, every time!!!!

pd.read_csv(filepath, sep="\t", header=False)

And I've been using this library almost every day for years

Re: Pandas 1.0

#65

Post by dm319 »

antipaul wrote: post #48
tel wrote: post #37

Earlier quoted context omitted.

I really, really dislike all the dtype wrangling and how those choices resonate throughout the API. I understand that a lot of work has been done to make that API "work", but in practice it feels like that effort would have been better avoided by changing expectations and interfaces. Now, to be clear, that's a hard problem. Heterogenous named bags of homogenous columns with a variety of data types, storage patterns,…

Hadley had the discipline to let go and start from scratch 2 if not 3 times before getting it perfect with `dplyr`. plyr came before and I think there was something else That’s how we got the amazing `dplyr` I think pandas is well liked by those who move from C++ or Java, but is disliked by those who move from R

I agree - R has performant and robust dataframe functions. dplyr is great for small-medium sized datasets, data.table seems to be really performant for larger sets.

Re: Pandas 1.0

#66

Post by byt143 »

spectramax wrote: post #47
appleiigs wrote: post #39

Earlier quoted context omitted.

I started using Julia recently. It seems like Julia has been able to take the good parts of Python and iron out the quirks. For example, I'm guessing the Julia DataFrame library is a knock off of Pandas, but the syntax more intuitive and concise - and I can remember it. For Julia itself, the syntax is very similar to Python but doesn't have the weird lambda functions. It has the Javascript style arrow for short anony…

Julia doesn't feel production ready at all. Its fine to mess around in notebooks but I would never recommend it for production use. Not even at a gunpoint. Debugger support is almost non-existent. Using Atom/Juno IDE is a D-grade experience. Julia offers little help to debug problems - errors are almost always without failure - completely tangent to what the real issue is. Julia takes forever to start, syntax was won…

To which version of Julia are you referring? All of those things are much improved now.

Re: Pandas 1.0

#67

Post by ololobus »

closed wrote: post #14

I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…

I dare to promote one StackOverflow question [1] about pandas I have tried to investigate and answer [2] half a year ago. And I was rather horrified by its internal complexity after digging into pandas source :)

OP was wondering, why pandas facing a strange overhead after each 100th iteration in some very specific case. There was a proposal about Python's GC, but it was not clear at all.

Finally, I have dived into pandas and found that it has a hard-coded constant == 100 (!) of a number of internal data storage blocks. After reaching this value it runs some consolidation routines [3], and they consume a lot of memory even leading to crash with memory error.

What was much more wondering, is that after changing this constant to some large value (1000000, actually it disables consolidation at all) reduces memory consumption dramatically! This consolidation seems to reduce storage and memory consumption, so I still do not know why the opposite happens and why it works well in all other cases.

[1] https://stackoverflow.com/questions/56690909/python-is-facin... [2] https://stackoverflow.com/a/56705419/978424 [3] https://github.com/pandas-dev/pandas/blob/761bceb77d44aa63b7...

Re: Pandas 1.0

#68

Post by milliams »

322849a8d87 wrote: post #30
_coveredInBees wrote: post #7

Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…

> yet the API is all over the place Agreed. In particular one might have hoped that 1.0 would fix indexing. .ix (deprecated), .loc, .iloc and "[" is an example of what people mean by saying the API is (a) a mess and (b) "deeply unpythonic". Shouldn't "[" be removed entirely if .loc and .iloc are recommended, given the odd and unpredictable edge cases with "["? > unless you use it regularly for a long period of time,…

That's almost exactly what they've started to do in 1.0. You say .ix is deprecated but in fact it was removed in 1.0. Now that 1.0 is out, they have a deprecation policy which will allow them to remove things like this.

Pandas' API might be a mess but that's partly because they're been really good about experimenting with the best way to do things for the last 10 (?) years. Adding newer alternatives to fiddly APIs etc. but never removing them. Now they can start the removals.

Re: Pandas 1.0

#69

Post by bart_spoon »

dm319 wrote: post #65
antipaul wrote: post #48

Earlier quoted context omitted.

Hadley had the discipline to let go and start from scratch 2 if not 3 times before getting it perfect with `dplyr`. plyr came before and I think there was something else That’s how we got the amazing `dplyr` I think pandas is well liked by those who move from C++ or Java, but is disliked by those who move from R

I agree - R has performant and robust dataframe functions. dplyr is great for small-medium sized datasets, data.table seems to be really performant for larger sets.

And now there is dtplyr, which simply creates a data.table back end with dplyr syntax on the front end.

Re: Pandas 1.0

#70

Post by appleiigs »

mirekrusin wrote: post #61
appleiigs wrote: post #59

Earlier quoted context omitted.

Yes, i was thinking about removing the Pandas code and using python iterables. The issue is I did a lot exploration with pandas, which it was good at. If I started with Julia, i wouldn’t need to refactor seeking performance by removing pandas, or for numba, dask, etc. For the existing project i’m thinking the switch to Julia + DataFrames library (despite it being a completely different language) is more of a 1:1 port…

I hear you can use python from Julia, have you tried this as, possibly temporary/transition, approach?

Yes, thanks. That ability definitely does give me a bit of comfort in case Julia is missing something. I plan not to use it though. When you use it, it’s actually running python, so same speed and such.
Post reply on HN