41–50 of 51 posts
It's a somewhat known tradeoff, you can streamline and make the format friendlier to do memcpy which this library targets, the more memcpys you do, the faster it is overall to decode. On highly compressible data lz4, snappy become faster. Snappy on level 2 has faster decompression speed But you have to pay the price that you need a slower encoding, because finding matches, putting restrictions on match lengths, putti…
Earlier quoted context omitted.
You probably meant MB/sec not GB.
No? Decompression is rated in GB/s per core now.
It used to say 2800 GB/s which is obviously bonkers.
Nice results, I will keep a watch on this! Would be interesting to see benchmarks vs Oodle compression, I think the most similar one is Selkie?
misa77 and oodle are both supported in turbobench (see: https://github.com/powturbo/turbobench), so it's easy to compare them.
Results on the silesia corpus on the same Intel setup described in the README, using turbobench:
codec decode ratio encode
misa77 -0 5309 MB/s 42.64% 58.2 MB/s
misa77 -1 4332 MB/s 39.65% 54.5 MB/s
selkie -1 3095 MB/s 43.48% 197 MB/s
selkie -2 3013 MB/s 41.80% 163 MB/s
selkie -3 3081 MB/s 39.58% 95.8 MB/s
selkie -4 3498 MB/s 36.95% 43.3 MB/s
selkie -5 2607 MB/s 33.58% 2.97 MB/s
lz4 2530 MB/s 47.60% 373 MB/s
cool, now do zstd!
Nice results, I will keep a watch on this! Would be interesting to see benchmarks vs Oodle compression, I think the most similar one is Selkie?
Yes, the most similar one is indeed Selkie. misa77 and oodle are both supported in turbobench (see: https://github.com/powturbo/turbobench ), so it's easy to compare them. Results on the silesia corpus on the same Intel setup described in the README, using turbobench: codec decode ratio encode misa77 -0 5309 MB/s 42.64% 58.2 MB/s misa77 -1 4332 MB/s 39.65% 54.5 MB/s selkie -1 3095 MB/s 43.48% 197 MB/s selkie -2 3013…
Earlier quoted context omitted.
Yes, the most similar one is indeed Selkie. misa77 and oodle are both supported in turbobench (see: https://github.com/powturbo/turbobench ), so it's easy to compare them. Results on the silesia corpus on the same Intel setup described in the README, using turbobench: codec decode ratio encode misa77 -0 5309 MB/s 42.64% 58.2 MB/s misa77 -1 4332 MB/s 39.65% 54.5 MB/s selkie -1 3095 MB/s 43.48% 197 MB/s selkie -2 3013…
Any chance you can throw lzop into that table just as a reference point?
codec decode ratio encode
misa77 -0 5302 MB/s 42.64% 58.7 MB/s
selkie -3 3078 MB/s 39.58% 97.0 MB/s
lzo1x -1 421 MB/s 47.48% 314 MB/s
memcpy 12135 MB/s 100.00% 11952 MB/sEarlier quoted context omitted.
No? Decompression is rated in GB/s per core now.
Parent fixed the error I pointed out since making my post by dividing by a thousand instead of changing the units so now my comment looks wrong. It used to say 2800 GB/s which is obviously bonkers.
Almost double the decompression speed, *and* a higher compression ratio than LZ4? That's very promising. I don't see much in your README that talks about how a developer would integrate misa into their code. I might suggest some basic code samples to help a developer integrate misa decode into their project. Congrats, looks like a cool project.