Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Siuba – A Dplyr Port to Python

github.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Siuba – A Dplyr Port to Python

#21

Post by ellisv »

closed wrote: post #16

Hey y'all, creator of siuba here--happy to answer any questions! One piece of context I try to bring into discussions is that the way I test and develop siuba is by livecoding data analyses for an hour [1]. I encounter a lot of arguments like "X is possible with pandas", but when I sit down with analysts in realistic settings (e.g. time constrained) it turns out X works in more limited ways then they thought [2][3].…

How's the performance? I'm certainly willing to give up a little computational performance for being able to write my code faster.

Re: Siuba – A Dplyr Port to Python

#22

Post by ellisv »

jphoward wrote: post #4

What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally. I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.…

> What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally. > I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.

I think I agree. Pandas has a lot of overhead/baggage that I don't want 90% of the time. Being able to chain _simple_ verbs on a data frame would be great -- something like mtcars.groupby(cyl).summarize(avg_hp = hp.mean())

Re: Siuba – A Dplyr Port to Python

#23

Post by ellisv »

disgruntledphd2 wrote: post #3
mangecoeur wrote: post #2

> A killer feature of siuba is that the same analysis code can be run on a local DataFrame, or a SQL source. Everyone thinks this is a good idea until they discover that SQL is not actually very portable, and any attempt to make it so neutralizes whatever benefits your SQL DB has. E.g. if you have a Postgres db, that has tons of excellent features that you actually want to use, sticking to a lowest-common-denominator…

I actually disagree, given that this is a dplyr port. dplyr has always had this feature, and I agree that it can make things difficult, but it can also provide a really, really useful abstraction over your SQL, especially if you need to change databases for whatever reason. In general, this actually looks like the best dplyr port I've seen, and may actually get me to do some exploratory analysis in Python.

I think this is a port of the dplyr syntax/verbs only. None of the dplyr C++ code is used, it's all syntactic sugar on top of Pandas. So any SQL translation is going to be similar to what Pandas does.

Re: Siuba – A Dplyr Port to Python

#24

Post by closed »

javitury wrote: post #7

Dplyr and data.table are two libraries that make R shine in terms of data manipulation. Python however has static analysis tools that are unavailable in R. I wonder if there is a data manipulation library in python that takes advantage of this. Looking at this library, it doesn't appear to use type hints. Other libraries, like pandas, have some basic support for typings but they are still far from being fully typed.

Siuba uses type hints to dispatch the appropriate versions of custom functions!

For example, siuba allows users to create custom functions using a thin wrapper around functools.singledispatch.

When deciding how to run...

    df >> filter(my_custom_func(_.x))
It requires that the return type be compatible with the backend being run (e.g. pandas, a SQL dialect).

Would be super interesting to try and lay out what would be needed to do static analysis via mypy. I think it'd require some plugins for singledispatch at least, probably some reworking things in ways myoy expects.

https://nbviewer.jupyter.org/github/machow/siuba/blob/master...

Re: Siuba – A Dplyr Port to Python

#25

Post by jphoward »

closed wrote: post #15
jphoward wrote: post #4

What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally. I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.…

Hey, author of siuba here, I totally agree that subclassing would be a natural choice in python. One challenge there is that users will often get a DataFrame back (e.g. from pd.read_csv), so it requires a lot of casting to the child class. Right now, a compromise I've been exploring is just attaching siuba's DF functions to a pandas DataFrame, e.g. df.siu_mutate(...). This seems to be what pandas wants people to do […

All good points! Great work on the project and thank you for working to make Pandas more friendly!

Re: Siuba – A Dplyr Port to Python

#26

Post by closed »

ellisv wrote: post #23
disgruntledphd2 wrote: post #3

Earlier quoted context omitted.

I actually disagree, given that this is a dplyr port. dplyr has always had this feature, and I agree that it can make things difficult, but it can also provide a really, really useful abstraction over your SQL, especially if you need to change databases for whatever reason. In general, this actually looks like the best dplyr port I've seen, and may actually get me to do some exploratory analysis in Python.

I think this is a port of the dplyr syntax/verbs only. None of the dplyr C++ code is used, it's all syntactic sugar on top of Pandas. So any SQL translation is going to be similar to what Pandas does.

siuba does the SQL translation :). pandas is used for local data, since it does a lot of optimization in c++, similar to dplyr's low level code.

Thanks for bringing this up--the docs could be clearer here

Re: Siuba – A Dplyr Port to Python

#27

Post by closed »

civilized wrote: post #11

This is, what, the fourth? Attempt to port dplyr to Python? Unclear why this one will be any more successful than the others

Hey, creator of siuba here. I think siuba's big advantage is that it can generate SQL code.

The architecture necessary to pull off executing either pandas or SQL also makes it very extensible (e.g. to spark or dask in the future :).

https://siuba.readthedocs.io/en/latest/key_features.html

Re: Siuba – A Dplyr Port to Python

#28

Post by closed »

ellisv wrote: post #21
closed wrote: post #16

Hey y'all, creator of siuba here--happy to answer any questions! One piece of context I try to bring into discussions is that the way I test and develop siuba is by livecoding data analyses for an hour [1]. I encounter a lot of arguments like "X is possible with pandas", but when I sit down with analysts in realistic settings (e.g. time constrained) it turns out X works in more limited ways then they thought [2][3].…

How's the performance? I'm certainly willing to give up a little computational performance for being able to write my code faster.

Using the experimental fast grouped pandas functions, it should run at the speed of optimized pandas code!

Since siuba functions just run on pandas DataFrames, you can always hand tune for performance, but imo most of the time pandas code runs slow it's because of something like .agg(lambda ...) somewhere.

There's an example with timings here:

https://siuba.readthedocs.io/en/latest/developer/pandas-grou...

Re: Siuba – A Dplyr Port to Python

#29

Post by disgruntledphd2 »

ellisv wrote: post #23
disgruntledphd2 wrote: post #3

Earlier quoted context omitted.

I actually disagree, given that this is a dplyr port. dplyr has always had this feature, and I agree that it can make things difficult, but it can also provide a really, really useful abstraction over your SQL, especially if you need to change databases for whatever reason. In general, this actually looks like the best dplyr port I've seen, and may actually get me to do some exploratory analysis in Python.

I think this is a port of the dplyr syntax/verbs only. None of the dplyr C++ code is used, it's all syntactic sugar on top of Pandas. So any SQL translation is going to be similar to what Pandas does.

Yeah, I probably used the wrong word, I didn't think it used the dplyr code, just that it looked much more like dplyr

Re: Siuba – A Dplyr Port to Python

#30

Post by closed »

ellisv wrote: post #22
jphoward wrote: post #4

What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally. I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr anyway.…

> What I think would be a nice approach would be someone creating a new Dataframe class that merely inherits from a Pandas dataframe but just implementing .filter(), .summarise(), .select() etc. as methods, using the dplyr syntax. If they return their own dataframes then chaining follows naturally. > I know that isn't how dplyr works, but it feels more Pythonic, and this solution isn't entirely analogous to dplyr any…

I'm still debating chaining vs piping, but you can do..

  from siuba import _
  from siuba.data import mtcars
  
  # mtcars is a pandas DataFrame

  mtcars \
    .groupby("cyl") \
    .siu_summarize(avg_hp=_.hp.mean())
Post reply on HN