51–60 of 80 posts
Assuming you want to use python. What’s the alternative to pandas? Kind of a brew up your own code kind of thing? Csv module I guess?
Dask is one option.
https://vaex.readthedocs.io/en/latest/
It has a cleaner, leaner API + the ability to use memory-mapped files.
Earlier quoted context omitted.
I started using Julia recently. It seems like Julia has been able to take the good parts of Python and iron out the quirks. For example, I'm guessing the Julia DataFrame library is a knock off of Pandas, but the syntax more intuitive and concise - and I can remember it. For Julia itself, the syntax is very similar to Python but doesn't have the weird lambda functions. It has the Javascript style arrow for short anony…
Julia doesn't feel production ready at all. Its fine to mess around in notebooks but I would never recommend it for production use. Not even at a gunpoint. Debugger support is almost non-existent. Using Atom/Juno IDE is a D-grade experience. Julia offers little help to debug problems - errors are almost always without failure - completely tangent to what the real issue is. Julia takes forever to start, syntax was won…
Although I spent a long time optimising numerical code with Numba the speedup I got (whilst significant) wasn’t really comparable to the speed of a Julia implementation.
It's great that the whole thing with extension arrays, custom types etc. has lead to this, but when the devs have, after 10+ years, the biggest chance for a backward incompatible change, this is the one to make. By making it optional, they are fixing it for the very few that know of its existence.
I love pandas and a sizeable part of my career depended on it - and while I don't use it anymore (partly because of the NULLs), I wish it the best and I hope there will be a future release with this breaking change.
Whenever one can use a utc epoch column for time indexed data in a raw numpy array instead, one should.
I've had to dive into the pandas code over the last year for a project [0], and my attitude has shifted dramatically from... * old attitude: why does pandas have to make things so hard * new attitude: pandas has a crazy difficult job I think this is most apparent in the functions that decide what "[d]type" a Block--the most basic thing that stores data in pandas--should be. https://github.com/pandas-dev/pandas/blob/4…
I really, really dislike all the dtype wrangling and how those choices resonate throughout the API. I understand that a lot of work has been done to make that API "work", but in practice it feels like that effort would have been better avoided by changing expectations and interfaces. Now, to be clear, that's a hard problem. Heterogenous named bags of homogenous columns with a variety of data types, storage patterns,…
That's a pretty succinct way of expressing one of panda's biggest pain points.
The API surface is huge because it heroically tries to handle all possible use cases. I do think it would've been significantly easier to develop, maintain and _use_ if pandas was more opinionated and offered a smaller set of very focused functions.
Great accomplishment and kudos to the dedicated maintainers. That being said, I've always had a love-hate relationship with pandas. It is a very powerful library and does a ton, but yet the API is all over the place and unless you use it regularly for a long period of time, it is almost impossible to get fluent with it. Every time I am away from it for a couple of months, I find even doing the most basic things to be…
Earlier quoted context omitted.
I started using Julia recently. It seems like Julia has been able to take the good parts of Python and iron out the quirks. For example, I'm guessing the Julia DataFrame library is a knock off of Pandas, but the syntax more intuitive and concise - and I can remember it. For Julia itself, the syntax is very similar to Python but doesn't have the weird lambda functions. It has the Javascript style arrow for short anony…
> I have a python/pandas script that take 3 days to run. Pandas .map() and .apply() get real slow on big datasets. I found it quicker to solve a problem with a million line dataset by just using base python iterables instead, so nothing needed to fit into my RAM and I didn't have to work with slow pandas mapping.
For the existing project i’m thinking the switch to Julia + DataFrames library (despite it being a completely different language) is more of a 1:1 port. In contrast would need to use brain power to build arrays or dicts to mimic Pandas (and probably get it wrong and introduce bugs)
There are quite a few complaints here about the interface being confusing and difficult to use, and I feel like some of this is due to there being significant differences between versions. I would love to read a medium-length online free tutorial on Pandas 1.0, but it seems like most of what turns on up google are short idiosyncratic tutorials on specific tasks in various versions.