81–90 of 146 posts
Arrow and SIMD, how would it work on Arm? I've had quite a success with Gravitons recently.
What is your question? ARM supports SIMD through NEON
Earlier quoted context omitted.
I dropped dplyr in favor of data.table and never looked back. https://github.com/eddelbuettel/gsir-te
Vanilla R got a bad name but once you understand the fundamentals it's quite good, fewer footguns than used to be there, and I find it easier to reason about than tidyverse.
It's better to separate benchmarking results for big data technologies and small DataFrame technologies.
Spark & Dask can perform computations on terabytes of data (thousands of Parquet files in parallel). Most of the other technologies in this article can only handle small datasets.
This is especially important for join benchmarking. There are different types of cluster computing joins (broadcast vs shuffle) and they should be benchmarked separately.
> At the time of writing this blog, Polars is the fastest DataFrame library in the benchmark second to R’s data.table, and Polars is top 3 all tools considered This is a very strange way to write “Polaris is the second fastest” but I guess that doesn’t grab headlines
I think it's mostly a nod to the fact that R's data.table blows everybody else out of the water by such a ridiculously wide margin. It's like a factor of 2 faster than the next fastest... So if you're writing a dataframe library as a hobby project, it's far less demotivating to use "all the other implementations" as your basis for comparison, at least initially.
I'm available for hookups,Plz check my personal website
> At the time of writing this blog, Polars is the fastest DataFrame library in the benchmark second to R’s data.table, and Polars is top 3 all tools considered This is a very strange way to write “Polaris is the second fastest” but I guess that doesn’t grab headlines
I think it's mostly a nod to the fact that R's data.table blows everybody else out of the water by such a ridiculously wide margin. It's like a factor of 2 faster than the next fastest... So if you're writing a dataframe library as a hobby project, it's far less demotivating to use "all the other implementations" as your basis for comparison, at least initially.
How does it compare to datafusion which is also a rust project that has dataframe support
What’s up with all the Dask benchmarks saying “internal error”? I expected at least some explanation in the post.
If you click through to the detailed benchmarks page ( https://h2oai.github.io/db-benchmark/ ). A lot of them are that it's running out of memory, a few of them are features that haven't been implemented yet. Inefficient use of memory is a problem I've seen with several projects that focus on scale out. All else being equal, they tend to use a lot more memory. This happens for various reasons, but a lot of it is the…
Earlier quoted context omitted.
What do you mean with smart disk spillover?
When the library attempts to load something from disk that doesn’t fit into memory, it’s transparently, and (usually) without extra intervention from the user, swaps to memory-mapping and chunking through the file(s). Particularly useful for when you’ve got a bunch of data that doesn’t fit in memory, but setting up a whole cluster is not worth the overhead (operationally or otherwise) and/or if you’ve already got a p…