I wrote one of the fastest DataFrame libraries
ritchievink.com
I wrote one of the fastest DataFrame libraries
1–10 of 146 posts
Re: I wrote one of the fastest DataFrame libraries
#2I've been intrigued about this library, and specifically the possibility about a Python workflow, but a fallback to rust if needed. I mean, I haven't really looked at what the interop is but should work, right?
It's not going to happen for now though because the project is still immature and there's zero documentation in Python from what I can see. But it's something in keeping a close eye on, I often work with R and C++ as a fallback when speed is paramount, but I think I'd rather replace C++ with Rust.
Re: I wrote one of the fastest DataFrame libraries
#3Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally
Re: I wrote one of the fastest DataFrame libraries
#4This is very cool. I'm happy to see the decision to use Arrow, which should make it almost trivially easy to transfer data into e.g. Julia, and maybe even to create bindings to Polar.
Re: I wrote one of the fastest DataFrame libraries
#5Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally
gzip, zcat, parallel... That sort of thing?
Re: I wrote one of the fastest DataFrame libraries
#6Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally
What do you mean with smart disk spillover?
Re: I wrote one of the fastest DataFrame libraries
#7Arrow and SIMD, how would it work on Arm? I've had quite a success with Gravitons recently.
Re: I wrote one of the fastest DataFrame libraries
#8What’s up with all the Dask benchmarks saying “internal error”? I expected at least some explanation in the post.
Re: I wrote one of the fastest DataFrame libraries
#9Not sure if anything exists but I wish something would do in memory compression + smart disk spillover. Sometimes I want to work with 5-10GB compressed data sets (usually log files) and decompressed that ends up being 10x (plus add data structure overhead). There's stuff like Apache Drill but it's more optimized for multi node than running locally
If you're not afraid of exotic languages, I encourage you to have a look at the APL ecosystem, and especially J and it's integrated columnar store Jd.
I have just embarked on an adventure to do just what you describe in... Racket. But it's nowhere to be seen yet.
I'm an epidemiologist and I've been wanting to make my own tools for a while, now. It'll be interesting to see how far I can go with Racket, which already includes many pieces of the puzzle.
Re: I wrote one of the fastest DataFrame libraries
#10Arrow and SIMD, how would it work on Arm? I've had quite a success with Gravitons recently.
What is your question? ARM supports SIMD through NEON