Shoco: a fast compressor for short strings
ed-von-schleck.github.io
Shoco: a fast compressor for short strings
1–10 of 20 posts
Re: Shoco: a fast compressor for short strings
#2I get negative compression percentage when I put words with "é" in the test box.
Re: Shoco: a fast compressor for short strings
#3I get negative compression percentage when I put words with "é" in the test box.
In default it doesn't work well with non-ASCII characters.
Re: Shoco: a fast compressor for short strings
#4I wonder what'd happen if you used this on base64 strings.
Re: Shoco: a fast compressor for short strings
#5I can't tell you how many times I've said to myself "if only these very short ASCII strings were even shorter!"
Re: Shoco: a fast compressor for short strings
#6Will test against smaz for our internal JSON compressed protocol. smaz compressed fine but was too slow. The ability to train the model sounds convincing.
Re: Shoco: a fast compressor for short strings
#7Look how well it can compress "fofofofofofofofofofofo".
50%
Look how well it can compress "ababababababababababab".
0%
Re: Shoco: a fast compressor for short strings
#8I have a background project of exploring how to compress SMILES strings, which is a notation for storing chemical information. For example, "C" is methane, "CC" is ethane, "C=C" is ethene, "CCO" is ethyl alcohol, "C1CCCCC1" is cyclohexane, and "c1ccccc1", which contains aromatic carbons, is benzene. The average length of a SMILES string for real-world molecules is about 50 characters.
I previously evaluated a special purpose tool which identifies the best n-grams and uses dynamic programming during encoding. That gets about 70% compression on SMILES string. I also tried the off-the-shelf femtozip which got about 60% compression but had more decompression overhead than I like.
Shoco, trained on 1,455,763 SMILES strings (average of 56 letters each), and tested with 100,000 strings from the training set, reports "average compression ratio: 47%".
Re: Shoco: a fast compressor for short strings
#9I get negative compression percentage when I put words with "é" in the test box.
In default it doesn't work well with non-ASCII characters. https://ed-von-schleck.github.io/shoco/#how-it-works
Between this (an ASCII-only compressor in 2015?) and the other aspects brought up here, it seems downright toylike.
Re: Shoco: a fast compressor for short strings
#10I can't tell you how many times I've said to myself "if only these very short ASCII strings were even shorter!"
At scale, and if you are paying for transmission costs, it can have a massive impact.