11–20 of 61 posts
Another way I've used image compression to identify cops that cover their body cameras while recording -- the filesize to length ratio reflects not much activity going on.
Sweet! I love clever information theory things like that. It goes the other way too. Given that LLMs are just lossless compression machines, I do sometimes wonder how much better they are at compressing plain text compared to zstd or similar. Should be easy to calculate... EDIT: lossless when they're used as the probability estimator and paired with something like an arithmetic coder.
I do not agree on the "lossless" adjective. And even if it is lossless, for sure it is not deterministic. For example I would not want a zip of an encyclopedia that uncompresses to unverified, approximate and sometimes even wrong text. According to this site : https://www.wikiwand.com/en/articles/Size%20of%20Wikipedia a compressed Wikipedia without medias, just text is ~24GB. What's the medium size of an LLM, 10 GB ?…
This looks like a nice rundown of how to do this with Python's zstd module. But, I'm skeptical of using compressors directly for ML/AI/etc. (yes, compression and intelligence are very closely related, but practical compressors and practical classifiers have different goals and different practical constraints). Back in 2023, I wrote two blog-posts [0,1] that refused the results in the 2023 paper referenced here (bad i…
Sweet! I love clever information theory things like that. It goes the other way too. Given that LLMs are just lossless compression machines, I do sometimes wonder how much better they are at compressing plain text compared to zstd or similar. Should be easy to calculate... EDIT: lossless when they're used as the probability estimator and paired with something like an arithmetic coder.
I do not agree on the "lossless" adjective. And even if it is lossless, for sure it is not deterministic. For example I would not want a zip of an encyclopedia that uncompresses to unverified, approximate and sometimes even wrong text. According to this site : https://www.wikiwand.com/en/articles/Size%20of%20Wikipedia a compressed Wikipedia without medias, just text is ~24GB. What's the medium size of an LLM, 10 GB ?…
With the usual interface it’s probably inefficient: giving just a prompt alone might not produce the output we need, or it might be larger than the thing we’re trying to compress. However, if we also steer the decisions along the way, we can probably give a small prompt that gets the LLM going, and tweak its decision process to get the tokens we want. We can then store those changes alongside the prompt. (This is a very hand-wavy concept, I know.)
(KL divergence of letter frequencies is the same thing as ratio of lengths of their Huffman-compressed bitstreams, but you don't need to do all this bit-twiddling for real just to count the letters)
The article views compression entirely through Python's limitations.
> gzip and LZW don’t support incremental compression
This may be true in the Python's APIs, but is not true about these algorithms in general.
They absolutely support incremental compression even in APIs of popular lower-level libraries.
Snapshotting/rewinding of the state isn't exposed usually (custom gzip dictionary is close enough in practice, but a dedicated API would reuse its internal caches). Algorithmically it is possible, and quite frequently used by the compressors themselves: Zopfli tries lots of what-if scenarios in a loop. Good LZW compression requires rewinding to a smaller symbol size and restarting compression from there after you notice the dictionary stopped being helpful. The bitstream has a dedicated code for this, so this isn't just possible, but baked into the design.
Earlier quoted context omitted.
I do not agree on the "lossless" adjective. And even if it is lossless, for sure it is not deterministic. For example I would not want a zip of an encyclopedia that uncompresses to unverified, approximate and sometimes even wrong text. According to this site : https://www.wikiwand.com/en/articles/Size%20of%20Wikipedia a compressed Wikipedia without medias, just text is ~24GB. What's the medium size of an LLM, 10 GB ?…
With a temperature of zero, LLM output will always be the same. Then it becomes a matter of getting it to output the exact replica of the input: if we can do that, it will always produce it, and the fact it can also be used as a bullshit machine becomes irrelevant. With the usual interface it’s probably inefficient: giving just a prompt alone might not produce the output we need, or it might be larger than the thing…
From what I understand, this is essentially how ts_zip (linked elsewhere) works.
Sweet! I love clever information theory things like that. It goes the other way too. Given that LLMs are just lossless compression machines, I do sometimes wonder how much better they are at compressing plain text compared to zstd or similar. Should be easy to calculate... EDIT: lossless when they're used as the probability estimator and paired with something like an arithmetic coder.
So yes, LLMs are nearly ideal text compressors, except for all the practical inconveniences of their size and speed (they can be reliably deterministic if you sacrifice parallel execution and some optimizations).
Sweet! I love clever information theory things like that. It goes the other way too. Given that LLMs are just lossless compression machines, I do sometimes wonder how much better they are at compressing plain text compared to zstd or similar. Should be easy to calculate... EDIT: lossless when they're used as the probability estimator and paired with something like an arithmetic coder.
The current leader on the Hutter Prize (http://prize.hutter1.net/) are all LLM based.
It can (slowly!!) compress a 1GB dump of Wikipedia to 106Mb
By comparison GZip can compress it to 321Mb
See https://mattmahoney.net/dc/text.html for the current leaderboard
The application of compressors for text statistics is fun, but it's a software equivalent of discovering that speakers and microphones are in principle the same device. (KL divergence of letter frequencies is the same thing as ratio of lengths of their Huffman-compressed bitstreams, but you don't need to do all this bit-twiddling for real just to count the letters) The article views compression entirely through Pytho…
I think it makes sense to explore it from practical standpoint, too. It’s in Python stdlib, and works reasonably well, so for some applications it might be good enough.
It’s also fairly easy to implement in other languages with zstd bindings, or even shell scripts:
$ echo 'taco burrito tortilla salsa guacamole cilantro lime' > /tmp/tacos.txt
$ zstd --train $(yes '/tmp/tacos.txt' | head -n 50) -o tacos.dict
[...snip]
$ echo 'racket court serve volley smash lob match game set' > /tmp/padel.txt
$ zstd --train $(yes '/tmp/padel.txt' | head -n 50) -o padel.dict
[...snip]
$ echo 'I ordered three tacos with extra guacamole' | zstd -D tacos.dict | wc -c
57
$ echo 'I ordered three tacos with extra guacamole' | zstd -D padel.dict | wc -c
60