11–20 of 51 posts
Interesting, but if you are not robust to corrupted/malicious data, it is really in a different class of algorithm and it is hard to compare speeds directly. From memory, 2505 MB/sec also sounds on the low side for LZ4 on a modern CPU?
You mean some kind of error detection? LZ4 doesn't have that.
- misa77's format may change unexpectedly as it's still v0.x.y.
- The decoder assumes that the input is a valid misa77 stream. Invalid input is UB and I offer no guarantees for whatever misa77 does in this case.
- It's been through some local fuzzing but is not hardened, so treat it as experimental.
It is slower than LZ4 on AArch64.
this is super interesting! im excited to give this a look this afternoon, since I specifically have wanted faster throughout for decompressing maps in a game engine.
It is slower than LZ4 on AArch64.
For reference, to test ARM64, I tested v0.1.0 on an M3 mac with this fork of lzbench: https://github.com/welcome-to-the-sunny-side/lzbench/tree/ad...
Here, lz4's decompression speed was far slower than misa77 and zxc. Results are here: https://github.com/welcome-to-the-sunny-side/misa77/blob/mai...
It's a very significant speedup in decompression speed (albeit with a compression speed slowdown as a trade-off), but what's the insight that makes it faster? What was the idea or approach behind it?
So, I couldn't see it in the readme, apologies if I missed it but why? It's a very significant speedup in decompression speed (albeit with a compression speed slowdown as a trade-off), but what's the insight that makes it faster? What was the idea or approach behind it?
Some concrete changes in the format are:
- match length per block is capped to 32
- distance to a match must be >= a fixed constant
- unlike lz4, tokens and literals have separate streams
- format of the token byte has been changed
The experimental compression modes (see readme) attempt to exploit this even further (but are even slower at compression). I define a "cost", which is a linear function of the branches induced by a compressed stream (this function serves as a proxy for decompression time), and then do a DP to minimise this cost.