Viewing profile — devin-petersohn

Funny enough, I actually just (2 weeks ago) added support for streaming from Pyspark to Polars/DuckDB/etc through Arrow PyCapsule. By streaming, I mean actually streaming, not coll…

The core abstraction of Modin is the dataframe algebra, published here: https://arxiv.org/pdf/2001.00888.pdf

I always appreciate blog posts like this, there are obviously cases where SQL shines, and in part I think the dataframe abstraction helps with filling a lot of the missing pieces t…

There are loads of things that are not possible or are very cumbersome to write in SQL, but that pandas and many other dataframe systems allow. Examples are dropping null values ba…

This blog post is a great way to learn about the theory underlying dataframes without reading a bunch of technical jargon from our academic paper[1]. AMA! [1] https://arxiv.org/pdf…

This was my PhD focus. We identified a core "dataframe algebra"[1] that encompasses all of pandas (and R/S data.frames): a total of 16 operators that cover all 600+ operators of pa…

I'm one the maintainers of Modin, so I can chime in here. Dataframes are the focus of my PhD thesis, and Modin started as my PhD project. Most of the differences come down to funct…