Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wrote one of the fastest DataFrame libraries

ritchievink.com

Post reply on HN
Print view View original

91–100 of 146 posts

Re: I wrote one of the fastest DataFrame libraries

#91

Post by ZephyrBlu »

rzk wrote: post #84
ohazi wrote: post #76

Earlier quoted context omitted.

I think it's mostly a nod to the fact that R's data.table blows everybody else out of the water by such a ridiculously wide margin. It's like a factor of 2 faster than the next fastest... So if you're writing a dataframe library as a hobby project, it's far less demotivating to use "all the other implementations" as your basis for comparison, at least initially.

Any idea what makes R's data.table so fast compared to the others?

I read somewhere else (Another comment I think) that it was a ground-up implementation taking a very performance orientated approach.

Basically it seemed like they really got in the weeds to make it super fast.

Re: I wrote one of the fastest DataFrame libraries

#92

Post by glogla »

MrPowers wrote: post #83

Looks like a cool project. It's better to separate benchmarking results for big data technologies and small DataFrame technologies. Spark & Dask can perform computations on terabytes of data (thousands of Parquet files in parallel). Most of the other technologies in this article can only handle small datasets. This is especially important for join benchmarking. There are different types of cluster computing joins (br…

Yeah, nobody uses Spark because they want to, they use it because from certain data size, there's nothing else.

Re: I wrote one of the fastest DataFrame libraries

#93

Post by Hendrikto »

> This directly shows a clear advantage over Pandas for instance, where there is no clear distinction between a float NaN and missing data, where they really should represent different things.

Not true anymore:

> Starting from pandas 1.0, an experimental pd.NA value (singleton) is available to represent scalar missing values. At this moment, it is used in the nullable integer, boolean and dedicated string data types as the missing value indicator.

> The goal of pd.NA is provide a “missing” indicator that can be used consistently across data types (instead of np.nan, None or pd.NaT depending on the data type).

(https://pandas.pydata.org/pandas-docs/stable/user_guide/miss...)

Re: I wrote one of the fastest DataFrame libraries

#94

Post by stabbles »

tkoolen wrote: post #73
SatvikBeri wrote: post #61

Earlier quoted context omitted.

What specifically are you thinking of? The non-const global variables stand out to me, but I'm not experienced enough tell whether that would make a large difference.

Non-const globals could be an issue, but it's possible it doesn't matter too much for this particular benchmark. I'm a little worried about taking compilation time (apart from precompilation) into account (would that also be done for C++ code?). But I must confess I maybe posted my comment a bit too soon, partially because of the time of day, partially because of the semicolons at the end of each line in the code, wh…

From the git history it seems like DataFrames.jl maintainers contributed at least some fixes to the scripts, so I guess that means they aren't opposed to it.

Re: I wrote one of the fastest DataFrame libraries

#95

Post by philshem »

ineedasername wrote: post #51

I'm surprised that data.table is so fast, and that pandas is so slow relative to it. It does explain why I've occasionally had memory issues on ~2GB data files when performing moderately complex functions. (to be fair, it's a relatively old Xeon w/ 12GB ram) I'll have to learn the nuances of data.table syntax now.

Are you using Pandas >= 1.0? I noticed a big speedup without changing my code.

Re: I wrote one of the fastest DataFrame libraries

#97

Post by ColFrancis »

ZephyrBlu wrote: post #91
rzk wrote: post #84

Earlier quoted context omitted.

Any idea what makes R's data.table so fast compared to the others?

I read somewhere else (Another comment I think) that it was a ground-up implementation taking a very performance orientated approach. Basically it seemed like they really got in the weeds to make it super fast.

R is from ~2000, while pandas started in 2011. Is it possible that the lack of compute power had an effect on the required performance characteristics?

Re: I wrote one of the fastest DataFrame libraries

#98

Post by sradman »

My naive interpretation - The canonical Apache Arrow implementation is written in C++ with multiple language bindings like PyArrow. The Rust bindings for Apache Arrow re-implemented the Arrow specification so it can be used as a pure Rust implementation. Andy Grove [1] built two projects on top of Rust-Arrow: 1. DataFusion, a query engine for Arrow that can optimize SQL-like JOIN and GROUP BY queries, and 2. Ballista, clustered DataFusion-like queries (vs. Dask and Spark). DataFusion was integrated into the Apache Arrow Rust project.

Ritchie Vink has introduced Polars that also builds upon Rust-Arrow. It offers an Eager API that is an alternative to PyArrow and a Lazy API that is a query engine and optimizer like DataFusion. The linked benchmark is focused on JOIN and GROUP BY queries on large datasets executed on a server/workstation-class machine (125 GB memory). This seems like a specialized use case that pushes the limits of a single developer machine and overlaps with the use case for a dedicated column store (like Redshift) or a distributed batch processing system like Spark/MapReduce.

Why Polars over DataFusion? Why Python bindings to Rust-Arrow rather than canonical PyArrow/C++? Is there something wrong with PyArrow?

[1] https://andygrove.io/projects/

Re: I wrote one of the fastest DataFrame libraries

#100

Post by ritchie46 »

sradman wrote: post #98

My naive interpretation - The canonical Apache Arrow implementation is written in C++ with multiple language bindings like PyArrow. The Rust bindings for Apache Arrow re-implemented the Arrow specification so it can be used as a pure Rust implementation. Andy Grove [1] built two projects on top of Rust-Arrow: 1. DataFusion, a query engine for Arrow that can optimize SQL-like JOIN and GROUP BY queries, and 2. Ballista…

Hi Author here,

Polars is not an alternative to PyArrow. Polars merely uses arrow as its in-memory representation of data. Similar to how pandas uses numpy.

Arrow provides the efficient data structures and some compute kernels, like a SUM, a FILTER, a MAX etc. Arrow is not a query engine. Polars is a DataFrame library on top of arrow that has implemented efficient algorithms for JOINS, GROUPBY, PIVOTs, MELTs, QUERY OPTIMIZATION, etc. (the things you expect from a DF lib).

Polars could be best described as an in-memory DataFrame library with a query optimizer.

Because it uses Rust Arrow, it can easily swap pointers around to pyarrow and get zero-copy data interop.

DataFusion is another query engine on top of arrow. They both use arrow as lower level memory layout, but both have a different implementation of their query engine and their API. I would say that DataFusion is more focused on a Query Engine and Polars is more focused an a DataFrame lib, but this is subjective.

Maybe its like comparing Rust Tokio vs Rust async-std. Just different implementations striving the same goal. (Only Polars and DataFusion can easily be mixed as they use the same memory structures).

Post reply on HN