Siuba – A Dplyr Port to Python
github.com
Siuba – A Dplyr Port to Python
1–10 of 32 posts
Re: Siuba – A Dplyr Port to Python
#2 > A killer feature of siuba is that the same analysis code can be run on a local DataFrame, or a SQL source.
Everyone thinks this is a good idea until they discover that SQL is not actually very portable, and any attempt to make it so neutralizes whatever benefits your SQL DB has.
E.g. if you have a Postgres db, that has tons of excellent features that you actually want to use, sticking to a lowest-common-denominator flavor of SQL basically nukes all that.
Re: Siuba – A Dplyr Port to Python
#3> A killer feature of siuba is that the same analysis code can be run on a local DataFrame, or a SQL source. Everyone thinks this is a good idea until they discover that SQL is not actually very portable, and any attempt to make it so neutralizes whatever benefits your SQL DB has. E.g. if you have a Postgres db, that has tons of excellent features that you actually want to use, sticking to a lowest-common-denominator…
I actually disagree, given that this is a dplyr port. dplyr has always had this feature, and I agree that it can make things difficult, but it can also provide a really, really useful abstraction over your SQL, especially if you need to change databases for whatever reason.
In general, this actually looks like the best dplyr port I've seen, and may actually get me to do some exploratory analysis in Python.
Re: Siuba – A Dplyr Port to Python
#4What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally.
I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.
This approach seems a little complicated, though I'm sure with some use I could learn to enjoy it.
Re: Siuba – A Dplyr Port to Python
#5[deleted]
Re: Siuba – A Dplyr Port to Python
#6kudos for doing this. Whilst I haven't used Suiba, dplyr in R is a very compelling way to do data munging. If Siuba brings the same thing to python, it's a very welcome addition to the ecosystem.
I know pipes split opinion; I'm definitely in the 'pro' camp. Chaining ops in a dataflow pipeline fits my mental model well.
Re: Siuba – A Dplyr Port to Python
#7Dplyr and data.table are two libraries that make R shine in terms of data manipulation.
Python however has static analysis tools that are unavailable in R. I wonder if there is a data manipulation library in python that takes advantage of this. Looking at this library, it doesn't appear to use type hints. Other libraries, like pandas, have some basic support for typings but they are still far from being fully typed.
Re: Siuba – A Dplyr Port to Python
#8This looks neat, the _ trick is similar to the Self [0] of fastcore (from fastai).
However many things are possible with vanilla pandas. I use it a lot for data munging, usually with the fluent interface (method chaining) style [1], e.g.:
df.loc[lambda f: ...].groupby(...).agg(["mean", "count"])
It also plays nicely with the black autoformatter.
Anonymous functions are verbose and limited in Python, but you can still do many things and use a regular function when a lambda won't do.
I guess one of the thing I need the most when doing data analysis is column name autocompletion, inside groupby, lambdas, for column selection...
I wonder if one could do it in IPython, similar to the string autocompletion of file/directory paths. Basically a parsing of dataframe column names in order to autocomplete strings.
[0]: https://fastcore.fast.ai/basics.html#Self-(with-an-uppercase...
Re: Siuba – A Dplyr Port to Python
#9This looks neat, the _ trick is similar to the Self [0] of fastcore (from fastai). However many things are possible with vanilla pandas. I use it a lot for data munging, usually with the fluent interface (method chaining) style [1], e.g.: df.loc[lambda f: ...].groupby(...).agg(["mean", "count"]) It also plays nicely with the black autoformatter. Anonymous functions are verbose and limited in Python, but you can still…
If the necessary information is there, as in you've mentioned a column name in a dataframe at least once, PyCharm will now do column name autocompletion. It's actually pretty solid in my experience.
Re: Siuba – A Dplyr Port to Python
#10Dplyr and data.table are two libraries that make R shine in terms of data manipulation. Python however has static analysis tools that are unavailable in R. I wonder if there is a data manipulation library in python that takes advantage of this. Looking at this library, it doesn't appear to use type hints. Other libraries, like pandas, have some basic support for typings but they are still far from being fully typed.
Don't forget ggplot2, another shiny tidyverse product. It has also been ported to python: https://plotnine.readthedocs.io/en/stable/