71–80 of 80 posts
"pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language." If anyone else is wondering what this is. (Source: project homepage
I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…
https://github.com/dylan-profiler/visions
The key idea is to allow abstraction of physical types (boolean, integers) by defining custom semantic types (URLs, paths, probabilities). The idea originated while working on pandas-profiling [0] and running into similar problems. We found this abstraction to be effective for many other downstream tasks, too, including compression and AutoML. More coming soon...
Earlier quoted context omitted.
I really, really dislike all the dtype wrangling and how those choices resonate throughout the API. I understand that a lot of work has been done to make that API "work", but in practice it feels like that effort would have been better avoided by changing expectations and interfaces. Now, to be clear, that's a hard problem. Heterogenous named bags of homogenous columns with a variety of data types, storage patterns,…
Agreed RE GroupBy being challenging, especially compared to dplyr. As I've worked on a port of dplyr to python over the past year, though, I've realized the dtype issue (like you said), indexes, and GroupBy being difficult are likely connected. Basically, * dplyr can chop up a dataframe into 50,000 groups and apply arbitrary functions to it--no problem. * custom pandas grouped applies are very slow There are basicall…
Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…
I started using Julia recently. It seems like Julia has been able to take the good parts of Python and iron out the quirks. For example, I'm guessing the Julia DataFrame library is a knock off of Pandas, but the syntax more intuitive and concise - and I can remember it. For Julia itself, the syntax is very similar to Python but doesn't have the weird lambda functions. It has the Javascript style arrow for short anony…
Could we collect some recommendations for really good books, online guides, tutorials, and recipes for current Pandas? There are quite a few complaints here about the interface being confusing and difficult to use, and I feel like some of this is due to there being significant differences between versions. I would love to read a medium-length online free tutorial on Pandas 1.0, but it seems like most of what turns on…
Earlier quoted context omitted.
I really, really dislike all the dtype wrangling and how those choices resonate throughout the API. I understand that a lot of work has been done to make that API "work", but in practice it feels like that effort would have been better avoided by changing expectations and interfaces. Now, to be clear, that's a hard problem. Heterogenous named bags of homogenous columns with a variety of data types, storage patterns,…
Hadley had the discipline to let go and start from scratch 2 if not 3 times before getting it perfect with `dplyr`. plyr came before and I think there was something else That’s how we got the amazing `dplyr` I think pandas is well liked by those who move from C++ or Java, but is disliked by those who move from R
Earlier quoted context omitted.
Julia doesn't feel production ready at all. Its fine to mess around in notebooks but I would never recommend it for production use. Not even at a gunpoint. Debugger support is almost non-existent. Using Atom/Juno IDE is a D-grade experience. Julia offers little help to debug problems - errors are almost always without failure - completely tangent to what the real issue is. Julia takes forever to start, syntax was won…
Just wanted to jump in here and say that for numerical computing Julia’s ecosystem of libraries are absolutely fantastic! DifferentialEquations.jl alone was worth switching to Julia! Although I spent a long time optimising numerical code with Numba the speedup I got (whilst significant) wasn’t really comparable to the speed of a Julia implementation.
The core developers are of Julia are very smart folks, they want to develop a great language that's fast and easy to use. They missed the opportunity to restrict syntax, provide useful exception message (just look at Rust! it is a thing of art when you get an exception, it is beautiful), and generally provide good documentation.
For example, just creating a Julia local registry requires significant overhead and time investment. Spinning up a registry should not take more than 30 mins.
All these aspects of Julia are prohibitive and in my opinion, Julia should not be used in any company or production use until perhaps version 2. People who have dealt with Julia issues will tell you the truth - not the academicians or researchers. The people that maintain infrastructure/maintenance support for Julia apps are almost ready to quit their jobs. It sucks so bad.
Earlier quoted context omitted.
Agreed RE GroupBy being challenging, especially compared to dplyr. As I've worked on a port of dplyr to python over the past year, though, I've realized the dtype issue (like you said), indexes, and GroupBy being difficult are likely connected. Basically, * dplyr can chop up a dataframe into 50,000 groups and apply arbitrary functions to it--no problem. * custom pandas grouped applies are very slow There are basicall…
I have been through this path of trying to figure out why pandas Groupby is slow. It is a big bottleneck when building packages on top of pandas.
I think at its core, the groupby issue is a really big problem, and am devoting most of this year to working on it. So if you ever want to pair to work on pandas / pandas wrapping libraries send me an email (link in profile)!
Earlier quoted context omitted.
Just wanted to jump in here and say that for numerical computing Julia’s ecosystem of libraries are absolutely fantastic! DifferentialEquations.jl alone was worth switching to Julia! Although I spent a long time optimising numerical code with Numba the speedup I got (whilst significant) wasn’t really comparable to the speed of a Julia implementation.
I love some of their libraries as you mentioned. Just that when the language itself is painful to write code and debug it, the whole value proposition is diminished. The core developers are of Julia are very smart folks, they want to develop a great language that's fast and easy to use. They missed the opportunity to restrict syntax, provide useful exception message (just look at Rust! it is a thing of art when you g…
Any specific examples?
I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…
I really, really dislike all the dtype wrangling and how those choices resonate throughout the API. I understand that a lot of work has been done to make that API "work", but in practice it feels like that effort would have been better avoided by changing expectations and interfaces. Now, to be clear, that's a hard problem. Heterogenous named bags of homogenous columns with a variety of data types, storage patterns,…
Aaand.. well, I guess it kind of is. But at the same time it isn't. Our biggest issues haven't been the complexity of what we've been implementing, but rather the insanity of the pandas interface. We've tried to keep it in mind that we're all inexperienced wrt. pandas, and it has definitely gotten better as we have gained experience, but that doesn't give us the hundreds of man-hours back we've spent trying to please pandas.
All the insane overloading of everything is awful and stupid. Give me 3 different ways to do 3 different things, not 9 ways to use 1 feature to do 1 thing.