Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wrote one of the fastest DataFrame libraries

ritchievink.com

Post reply on HN
Print view View original

71–80 of 146 posts

Re: I wrote one of the fastest DataFrame libraries

#71

Post by qchris »

wealthyyy wrote: post #64
lordgroff wrote: post #2

I've been intrigued about this library, and specifically the possibility about a Python workflow, but a fallback to rust if needed. I mean, I haven't really looked at what the interop is but should work, right? It's not going to happen for now though because the project is still immature and there's zero documentation in Python from what I can see. But it's something in keeping a close eye on, I often work with R and…

Rust projects takes longer. If memory safety is not a concern, I'd advice stick to Modern C++.

This feels like a gross generalization that's not applicable in many situations, and is immensely dependent on each individual person and situation.

I can write non-trivial performant code in Rust, including bindings across a C FFI much faster than I can weave together the equivalent code and build scripts in C++. Memory safety isn't the only thing Rust brings to the table. I sometimes don't because C++'s ecosystem is far developed for a certain application and it's not worth it for that particular situation. As with most things, it's about trade-offs.

Re: I wrote one of the fastest DataFrame libraries

#72

Post by sonthonax »

sandGorgon wrote: post #65

"Polars is based on the Rust native implementation Apache Arrow. Arrow can be seen as middleware software for DBMS, query engines and DataFrame libraries. Arrow provides very cache-coherent data structures and proper missing data handling." This is super cool. Anyone know if Pandas is also planning to adopt Arrow ?

It’ll probably never be fully comparable because Pandas can represent python objects and nulls (badly). However, for the most part Arrow and Numpy are compatible. There’s no overhead in converting an arrow data structure into a Numpy one.

Re: I wrote one of the fastest DataFrame libraries

#73

Post by tkoolen »

SatvikBeri wrote: post #61
tkoolen wrote: post #54

[note: see more nuanced comment below] The Julia benchmark two links deep at https://github.com/h2oai/db-benchmark doesn't follow even the most basic performance tips listed at https://docs.julialang.org/en/v1/manual/performance-tips/ .

What specifically are you thinking of? The non-const global variables stand out to me, but I'm not experienced enough tell whether that would make a large difference.

Non-const globals could be an issue, but it's possible it doesn't matter too much for this particular benchmark. I'm a little worried about taking compilation time (apart from precompilation) into account (would that also be done for C++ code?). But I must confess I maybe posted my comment a bit too soon, partially because of the time of day, partially because of the semicolons at the end of each line in the code, which made me quickly think the benchmark writer was using Julia for the first time. While I have a good amount of experience with Julia, I don't have that much experience with DataFrames.jl itself, so I don't know for sure whether the reported benchmark times are reasonable or not.

Re: I wrote one of the fastest DataFrame libraries

#75

Post by timClicks »

machinelabo wrote: post #53
SatvikBeri wrote: post #39

Earlier quoted context omitted.

I started using Julia in December, DataFrames are in a sort of weird place because they're so much less necessary compared to e.g. Python. In Julia, you could just use a dict of arrays and get most of the benefits, thanks to libraries like Query.jl and Tables.jl. Thus the ecosystem is a lot more spread out. I actually use DataFrames much less than I used to in Python. This is mostly good, because you can apply the sa…

Those arguments apply to Python as well. There is nothing special about Julia that warrants your arguments.

I disagree. Python's data science community is strongly clustered around pandas, even though it's possible to use other approaches

Re: I wrote one of the fastest DataFrame libraries

#76

Post by ohazi »

qeternity wrote: post #70

> At the time of writing this blog, Polars is the fastest DataFrame library in the benchmark second to R’s data.table, and Polars is top 3 all tools considered This is a very strange way to write “Polaris is the second fastest” but I guess that doesn’t grab headlines

I think it's mostly a nod to the fact that R's data.table blows everybody else out of the water by such a ridiculously wide margin. It's like a factor of 2 faster than the next fastest...

So if you're writing a dataframe library as a hobby project, it's far less demotivating to use "all the other implementations" as your basis for comparison, at least initially.

Re: I wrote one of the fastest DataFrame libraries

#77

Post by de6u99er »

I tried running your code via docker-compose. After some building time, none of the notebooks in examples-folder worked.

The notebook with the title "10 minutes to pypolars" was missing the pip command which I had to add to your Dockerfile (actually python-pip3). After rebuilding the whole thing and restarting the notebook, I had to change "!pip" to "!pip3" (was to lazy to add an alias) in the first code-cell which installed all dependencies after running. All the other cells resulted in errors.

I suggest to focus on stability and reproducibility first and then on performance.

Re: I wrote one of the fastest DataFrame libraries

#79

Post by ritchie46 »

de6u99er wrote: post #77

I tried running your code via docker-compose. After some building time, none of the notebooks in examples-folder worked. The notebook with the title "10 minutes to pypolars" was missing the pip command which I had to add to your Dockerfile (actually python-pip3). After rebuilding the whole thing and restarting the notebook, I had to change "!pip" to "!pip3" (was to lazy to add an alias) in the first code-cell which i…

Author here. These examples and docker-composes files are heavily outdated. Please take a look at the docs for up to date examples.

P.S. I do what I can to keep things up to date, but only have the time I have.

Re: I wrote one of the fastest DataFrame libraries

#80

Post by hugh-avherald »

orhmeh09 wrote: post #32
rcthompson wrote: post #28

Earlier quoted context omitted.

dplyr and related packages use the existing R data frame class. (A "tibble" is just a regular R data frame under the hood.) This means that it inherits all the performance characteristics of regular R data frames. data.table is a completely separate implementation of a data structure that is functionally similar to a data frame but designed from the ground up for efficiency, though with some compromises, such as esch…

Oh, I know that, I use it daily and I’ve read some of its source code. I’m just astonished that the best-performing data frame library in the world is developed in R and it outperforms engines written with million/billion dollar companies behind it.

data.table is written primarily in C. But R happens to have a very good package system and a very good interface to C code.

And Matt Dowle has bled for that C code.

Post reply on HN