Assuming you want to use python. What’s the alternative to pandas? Kind of a brew up your own code kind of thing? Csv module I guess?
Well, depends on what you want to achieve.
Pandas 1.0
31–40 of 80 posts
Re: Pandas 1.0
#32I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…
There are always choices to make. In this case, I would much prefer to let the data be treated as is, i.e., no silent casting of np.nan to the string "nan". When dealing with numerical data, a string "nan" is rarely useful. But when you need it, you can still create a data series with a string "nan" using pd.Series([str(np.nan), "a"])
So you want the behavior that Pandas has (i.e., no silent casting of np.nan to the string "nan").
Re: Pandas 1.0
#33Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…
PyTorch is a project primarily funded/developed by Facebook/FAIR. Pandas is fully an open-source project, without corporate control.
Corporate control means tighter development schedules and consistent API's. It also means that if you don't like the path FAIR has chosen, too bad. As a result, there's multiple competing options in the deep learning space: Tensorflow (Google), MXNet (Amazon), CNTK (Microsoft), Paddle (Baidu), etc.
On the other hand, Pandas is something for everyone. The lack of opinioniation means that it can be easily adopted anywhere. Can you imagine what data science/analysis would feel like with multiple low-level Pandas competitors, from different corporations? Each one would feel consistent, but none would work together (and imagine building an ML platform which supported multiple dataframe sources).
I do sometimes miss working in R - yes, R takes flexibility to a fault, but there's a consistent set of primitives that mostly get reused. Perhaps R gives off that impression because of the work done by Hadley and others to build tooling according to the tidyverse principles. I wonder if Julia will combine the best of these worlds in the future.
Re: Pandas 1.0
#34Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…
The danger with Pandas’ bizarre API is that it isn’t obvious when you are doing something stupid, because the right way to say things also looks weird and unpythonic.
On the other hand, I’ve gotten tremendous value from it, and I can’t aggressively criticize an open source project I can use without paying.
Re: Pandas 1.0
#35Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…
> yet the API is all over the place Agreed. In particular one might have hoped that 1.0 would fix indexing. .ix (deprecated), .loc, .iloc and "[" is an example of what people mean by saying the API is (a) a mess and (b) "deeply unpythonic". Shouldn't "[" be removed entirely if .loc and .iloc are recommended, given the odd and unpredictable edge cases with "["? > unless you use it regularly for a long period of time,…
Definitely agree the API can be tightened up, but just for fun and perspective take a look at pre pandas data wrangling tools like sas.
Like Example x3 vs x5 in their docs on the sum function:
http://support.sas.com/documentation/cdl/en/lrdict/64316/HTM...
Re: Pandas 1.0
#36I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…
I think another area pandas has done a lot of work on is with datetimes. Numpy's datetime objects are pretty deficient when you need to perform computations / data wrangling with them and utilizing python's native datetime objects would slow things down a decent bit. So they have done a lot of work to create their own datetime implementation that helps a lot when dealing with tabular data and performing date/time bas…
I use Julia and this is one area where Pandas still kicks the Julia ecosystem's proverbial ass: awesome support for working with nanosecond precision epoch timestamps.
Re: Pandas 1.0
#37I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…
I really, really dislike all the dtype wrangling and how those choices resonate throughout the API. I understand that a lot of work has been done to make that API "work", but in practice it feels like that effort would have been better avoided by changing expectations and interfaces.
Now, to be clear, that's a hard problem. Heterogenous named bags of homogenous columns with a variety of data types, storage patterns, and ideas about missingness isn't an easy domain... but instead of just trying to make everything work through hammering 6+ semi-coherent interfaces (indices, databases, mutability, immutability/chaining, numpy, dataframes) together, I'd be willing to pay a lot more in verbosity and explicitness for something simple.
pd.Series(str, [np.nan, "a"]) => ["nan", "a"] # or even an exception!
pd.Series(nullable(str), [np.nan, "a"]) => [nan, "a"]
I'd chalk this all up to just being "hard", but at the same time I can go pick up R's dplyr library and get a very nice existence proof of how a nice interface could work. Not to say dplyr has it all figured out, but it's a night-and-day improvement to Pandas.
Pandas is great. It makes doing data science in Python so vastly much less of a chore than working with straight Numpy. It steals some great ideas and tries out a few interesting ones of its own... but it is far from a joy to work with.
Re: Pandas 1.0
#38Earlier quoted context omitted.
There are always choices to make. In this case, I would much prefer to let the data be treated as is, i.e., no silent casting of np.nan to the string "nan". When dealing with numerical data, a string "nan" is rarely useful. But when you need it, you can still create a data series with a string "nan" using pd.Series([str(np.nan), "a"])
So you want the behavior that Pandas has (i.e., no silent casting of np.nan to the string "nan").
The point I wanted to make was that string "nan" is usually not as useful as floating point NaN when dealing with numerical data. Therefore, as default behavior it is acceptable, if one has to make a choice.
Re: Pandas 1.0
#39Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…
I started using Julia recently. It seems like Julia has been able to take the good parts of Python and iron out the quirks. For example, I'm guessing the Julia DataFrame library is a knock off of Pandas, but the syntax more intuitive and concise - and I can remember it.
For Julia itself, the syntax is very similar to Python but doesn't have the weird lambda functions. It has the Javascript style arrow for short anonymous functions and the Ruby style "do" for longer functions. And finally, Julia is fast. I have a python/pandas script that take 3 days to run. Moving it over to Julia now.
Re: Pandas 1.0
#40I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…
I really, really dislike all the dtype wrangling and how those choices resonate throughout the API. I understand that a lot of work has been done to make that API "work", but in practice it feels like that effort would have been better avoided by changing expectations and interfaces. Now, to be clear, that's a hard problem. Heterogenous named bags of homogenous columns with a variety of data types, storage patterns,…
Agreed RE GroupBy being challenging, especially compared to dplyr.
As I've worked on a port of dplyr to python over the past year, though, I've realized the dtype issue (like you said), indexes, and GroupBy being difficult are likely connected. Basically,
* dplyr can chop up a dataframe into 50,000 groups and apply arbitrary functions to it--no problem.
* custom pandas grouped applies are very slow
1. creating an index for each subgroup is slow (will not be a RangeIndex)
2. initializing a series for each group is slow (mostly due to type inference being re-run; could be avoided)
3. AFAIK more type inference is run when concatenating results
I wrote a bit on how I tried to work around that, to enable fast dplyr-like syntax over grouped data in python. Would definitely be interested in your take! There are other libraries, like ibis that do a good job with it, too!
https://siuba.readthedocs.io/en/latest/developer/pandas-grou...