Compressor "firn" improves ZSTD by 13%

#1

Post by seccode » Tue, Nov 26, 2024, 2:53 AM UTC

Compressor "firn" improves ZSTD by 13%
github.com

Re: Compressor "firn" improves ZSTD by 13%

#2

Post by themerone » Tue, Nov 26, 2024, 2:57 AM UTC

This works by text substitution, I imagine that you could get similar or better gains using zstd custom dictionaries.

Re: Compressor "firn" improves ZSTD by 13%

#3

Post by seccode » Tue, Nov 26, 2024, 2:59 AM UTC

themerone wrote: post #2

This works by text substitution, I imagine that you could get similar or better gains using zstd custom dictionaries.

This is a direct comparison to zstd with a dictionary actually

Re: Compressor "firn" improves ZSTD by 13%

#4

Post by seccode » Tue, Nov 26, 2024, 3:00 AM UTC

seccode wrote: post #3

themerone wrote: post #2

This works by text substitution, I imagine that you could get similar or better gains using zstd custom dictionaries.

This is a direct comparison to zstd with a dictionary actually

Methodology for comparison: train zstd dictionary on enwik9. Then build my dictionary as most common words in enwik9. Mine does 13% better because of the way I discovered how you can generate dictionary replacement symbols