21–30 of 51 posts
So, I couldn't see it in the readme, apologies if I missed it but why? It's a very significant speedup in decompression speed (albeit with a compression speed slowdown as a trade-off), but what's the insight that makes it faster? What was the idea or approach behind it?
Earlier quoted context omitted.
Source?
Just tested a few minutes ago on the ClickBench dataset. Overall quite good, but, depending on particular columns, most of the time slower - e.g., ~2.0 vs ~2.8 GB/sec on Graviton 4 machine in AWS.
But you have to pay the price that you need a slower encoding, because finding matches, putting restrictions on match lengths, putting things in different streams have costs you need to pay upfront.
Anyway good work, there is probably a need for that.
// Currently own Google's snappy and do compression at Google
P.S. if you want better snappy's results, compile with clang.
P.S.S you can optimize aarch64 speed by movemasks from shrn instruction. https://developer.arm.com/community/arm-community-blogs/b/se...
So, I couldn't see it in the readme, apologies if I missed it but why? It's a very significant speedup in decompression speed (albeit with a compression speed slowdown as a trade-off), but what's the insight that makes it faster? What was the idea or approach behind it?
So it's essentially "LZ4 unshackled". I've made several modifications to the LZ4 format, most of which are in service of eliminating branches/making them more predictable, and making decompression very friendly to out-of-order cores by hiding false data dependencies behind a rarely taken branch (similar to this: https://news.ycombinator.com/item?id=48889148 ). Some concrete changes in the format are: - match length p…
And i do not mean this in a flippant way, as how to harden with speed in mind might alter how to design the format and the codec.
Earlier quoted context omitted.
So it's essentially "LZ4 unshackled". I've made several modifications to the LZ4 format, most of which are in service of eliminating branches/making them more predictable, and making decompression very friendly to out-of-order cores by hiding false data dependencies behind a rarely taken branch (similar to this: https://news.ycombinator.com/item?id=48889148 ). Some concrete changes in the format are: - match length p…
How much of the speed-up is attributed to not hardening the code? And i do not mean this in a flippant way, as how to harden with speed in mind might alter how to design the format and the codec.
It's a somewhat known tradeoff, you can streamline and make the format friendlier to do memcpy which this library targets, the more memcpys you do, the faster it is overall to decode. On highly compressible data lz4, snappy become faster. Snappy on level 2 has faster decompression speed But you have to pay the price that you need a slower encoding, because finding matches, putting restrictions on match lengths, putti…
I haven't implemented any ARM-specific dispatch yet (there are currently no NEON paths for vector ops either, and we instead trust the compiler to autovectorize), and will do so for upcoming versions.
Earlier quoted context omitted.
Just tested a few minutes ago on the ClickBench dataset. Overall quite good, but, depending on particular columns, most of the time slower - e.g., ~2.0 vs ~2.8 GB/sec on Graviton 4 machine in AWS.
You probably meant MB/sec not GB.
So, I couldn't see it in the readme, apologies if I missed it but why? It's a very significant speedup in decompression speed (albeit with a compression speed slowdown as a trade-off), but what's the insight that makes it faster? What was the idea or approach behind it?
So it's essentially "LZ4 unshackled". I've made several modifications to the LZ4 format, most of which are in service of eliminating branches/making them more predictable, and making decompression very friendly to out-of-order cores by hiding false data dependencies behind a rarely taken branch (similar to this: https://news.ycombinator.com/item?id=48889148 ). Some concrete changes in the format are: - match length p…