Pretty impressed with the data.table benchmarks. The syntax is a little weird and takes getting used to but once you have the basics it’s a great tool.
I wrote one of the fastest DataFrame libraries
11–20 of 146 posts
Re: I wrote one of the fastest DataFrame libraries
#12Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally
I feel like the HDF5 libraries could be helpful here if you could figure out how to get compatible compression.
Re: I wrote one of the fastest DataFrame libraries
#13Pretty impressed with the data.table benchmarks. The syntax is a little weird and takes getting used to but once you have the basics it’s a great tool.
I use it a lot but it really breaks the tidyverse, which makes using R actually enjoyable. Why aren’t these other libraries (not in R; I’m talking the others in the benchmark) consistently as fast as data.table? Are the programmers of data.table just that much better?
Re: I wrote one of the fastest DataFrame libraries
#14Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally
Dask is popular for this purpose. No memory compression, but supports disk spillage as well as distributed dataframes.
Re: I wrote one of the fastest DataFrame libraries
#15Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally
If youre doing OLAP style queries you should look at DuckDB, it's hella fast and supports out-of-memory compute (it's not exactly "smart" but it handles spillover)
Re: I wrote one of the fastest DataFrame libraries
#16I would love to learn the details of building a Python wrapper on Rust code like you did with pypolars.
Re: I wrote one of the fastest DataFrame libraries
#17Pretty impressed with the data.table benchmarks. The syntax is a little weird and takes getting used to but once you have the basics it’s a great tool.
I use it a lot but it really breaks the tidyverse, which makes using R actually enjoyable. Why aren’t these other libraries (not in R; I’m talking the others in the benchmark) consistently as fast as data.table? Are the programmers of data.table just that much better?
I dropped dplyr in favor of data.table and never looked back.
Re: I wrote one of the fastest DataFrame libraries
#18Pretty impressed with the data.table benchmarks. The syntax is a little weird and takes getting used to but once you have the basics it’s a great tool.
Me too: I've tended to let the database do a lot of heavy lifting before I bring data in. Maybe I don't actually need to do that.
Re: I wrote one of the fastest DataFrame libraries
#19What’s up with all the Dask benchmarks saying “internal error”? I expected at least some explanation in the post.
These are single machine benchmarks.
Re: I wrote one of the fastest DataFrame libraries
#20Still mind boggling to me how amazing data.table is