This is, what, the fourth? Attempt to port dplyr to Python? Unclear why this one will be any more successful than the others
Siuba – A Dplyr Port to Python
11–20 of 32 posts
Re: Siuba – A Dplyr Port to Python
#12This looks neat, the _ trick is similar to the Self [0] of fastcore (from fastai). However many things are possible with vanilla pandas. I use it a lot for data munging, usually with the fluent interface (method chaining) style [1], e.g.: df.loc[lambda f: ...].groupby(...).agg(["mean", "count"]) It also plays nicely with the black autoformatter. Anonymous functions are verbose and limited in Python, but you can still…
If the necessary information is there, as in you've mentioned a column name in a dataframe at least once, PyCharm will now do column name autocompletion. It's actually pretty solid in my experience.
Didn't know pycharm would do that... thanks for the info! Yeah the best things about dplyr in opinion are 1) less verbose than pandas 2) much better autocomplete.
Re: Siuba – A Dplyr Port to Python
#13This looks neat, the _ trick is similar to the Self [0] of fastcore (from fastai). However many things are possible with vanilla pandas. I use it a lot for data munging, usually with the fluent interface (method chaining) style [1], e.g.: df.loc[lambda f: ...].groupby(...).agg(["mean", "count"]) It also plays nicely with the black autoformatter. Anonymous functions are verbose and limited in Python, but you can still…
IPython already does tab completion of data frame column names. E.G., `df[“col` will do what you’d hope.
Re: Siuba – A Dplyr Port to Python
#14Neat. I've been working on my own "piped-Codd" style system I call the "data algebra" https://github.com/WinVector/data_algebra
I use method chaining as the composing notation.
Re: Siuba – A Dplyr Port to Python
#15What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally. I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.…
Hey, author of siuba here, I totally agree that subclassing would be a natural choice in python. One challenge there is that users will often get a DataFrame back (e.g. from pd.read_csv), so it requires a lot of casting to the child class.
Right now, a compromise I've been exploring is just attaching siuba's DF functions to a pandas DataFrame, e.g. df.siu_mutate(...). This seems to be what pandas wants people to do [1]! One obstacle here is that the DataFrame has 300+ methods, which can be overwhelming to learners.
I've spent a lot of time wondering whether the piping syntax feels like too much vs chaining. It's still an open question in my mind, so it's really helpful to hear what feels most natural!
https://pandas.pydata.org/pandas-docs/stable/development/ext...
Re: Siuba – A Dplyr Port to Python
#16Hey y'all, creator of siuba here--happy to answer any questions!
One piece of context I try to bring into discussions is that the way I test and develop siuba is by livecoding data analyses for an hour [1]. I encounter a lot of arguments like "X is possible with pandas", but when I sit down with analysts in realistic settings (e.g. time constrained) it turns out X works in more limited ways then they thought [2][3].
I'm a big fan of pandas though. It's what siuba is built on!
[1]: https://m.youtube.com/c/chowthedog
[2]: https://mchow.com/posts/2020-02-11-dplyr-in-python/
[3]: https://siuba.readthedocs.io/en/latest/developer/pandas-grou...
Re: Siuba – A Dplyr Port to Python
#17This is, what, the fourth? Attempt to port dplyr to Python? Unclear why this one will be any more successful than the others
This one looks much, much better than the others, as a long-time user of dplyr who's now a refugee in the Python universe.
Re: Siuba – A Dplyr Port to Python
#18What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally. I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.…
Hey, author of siuba here, I totally agree that subclassing would be a natural choice in python. One challenge there is that users will often get a DataFrame back (e.g. from pd.read_csv), so it requires a lot of casting to the child class. Right now, a compromise I've been exploring is just attaching siuba's DF functions to a pandas DataFrame, e.g. df.siu_mutate(...). This seems to be what pandas wants people to do […
What about monkey patching the class at runtime?
Edit: oh that's pretty much what the linked decorators do.
Re: Siuba – A Dplyr Port to Python
#19This looks neat, the _ trick is similar to the Self [0] of fastcore (from fastai). However many things are possible with vanilla pandas. I use it a lot for data munging, usually with the fluent interface (method chaining) style [1], e.g.: df.loc[lambda f: ...].groupby(...).agg(["mean", "count"]) It also plays nicely with the black autoformatter. Anonymous functions are verbose and limited in Python, but you can still…
Hey, thanks for pointing out Self--I definitely need to dig into fastcore more!
One motivation for developing siuba is that the grouped agg you show requires users specify only one operation on one column.
E.g.
1. Calculate mean of x
However, common operations like demeaning a column are multiple operations:
1. Calculate mean of x
2. Subtract result of (1) from x
In siuba you can just write mutate(res = _.x -_.x.mean()). This isn't possible from something like gdf.x.agg("mean"), and from what I can tell deeply confusing to analysts :/.
In vanilla pandas I really like to use the chaining method you laid out, and siuba to me is mostly a utility library for making the approach a little more succinct / performant[1].
siuba has experimental autocompletion (thanks to Tim Mastny!), and there's a pretty hefty technical write up on how it uses IPython machinery for that in siuba's architectural desicion record folder[2].
[1]: https://siuba.readthedocs.io/en/latest/developer/pandas-grou...
[2]: https://github.com/machow/siuba/blob/master/examples/archite...
Re: Siuba – A Dplyr Port to Python
#20This looks neat, the _ trick is similar to the Self [0] of fastcore (from fastai). However many things are possible with vanilla pandas. I use it a lot for data munging, usually with the fluent interface (method chaining) style [1], e.g.: df.loc[lambda f: ...].groupby(...).agg(["mean", "count"]) It also plays nicely with the black autoformatter. Anonymous functions are verbose and limited in Python, but you can still…
IPython already does tab completion of data frame column names. E.G., `df[“col ` will do what you’d hope.
As long as there isn't a space in the column name. This is riding on a Pandas trick of making the column name accessible as an attribute of the dataframe, which breaks down when there's a space.