31–38 of 38 posts
I wonder what if anything has changed ever since this article. Is llm-based compression more mainstream?
I suspect that LLMs wouldn't be ideal to use as compressors, because they are large, consume a lot of resources, and are constantly changing. You need the model to produce exactly the same output at encoding and decoding time, or else you get gibberish.
Earlier quoted context omitted.
I think this is an analogy that's been taken far too far. The output of intelligence just isn't compression, that's memorization. The role of intelligence is to generate novelty. It's true that LLMs do something that looks very compression like in their weights, but it is lossy, and it has to be--if you're not lossy, you've overfitted the corpus, and that's bad. Post-training takes this even further, because you're n…
LLM compression doesn't necessarily have to be lossy. You can use the fact that LLMs predict P(next token | existing tokens) to losslessly and efficiently compress arbitrary token sequences. This idea is closely related to arithmetic coding.
Matt Mahoney is one of the best when it comes to compression. He is retired now.
Isn’t the idea of AI precisely to find universal compression from arbitrary input data, at least with LLMs?
Which is why in this book the title of a paragraph is "Compression is an artificial intelligence problem". However, I believe that is less useful to think that AI "finds an universal compression", than to think that the training of an AI model has the purpose to find a specific lossy data compression method, which is close to optimal for the input data that constitutes the training data set. One could consider the tr…
If you go and (re)read what he writes in relation to AI, which I just did, it's about exclusion. He excludes "Universal Compression" as impossible, Kolmogorov compression as uncomputable, and then he gets to Artificial Intelligence. Artificial Intelligence is an appropriate way to model data, since data is created by humans with human intelligence. And, AI doesn't violate mathematics the way Universal Compression and Kolmogorov solutions do. So therefore, Artificial Intelligence is what's left. That seems to be the argument.
does anyone have any sources to read about ai-based compression? I remember hearing a lot about "compression is a lot about prediction", but I don't remember reading any practical result
Earlier quoted context omitted.
No, it's not for lossy compression only. An LLM can give you a probability distribution for the next token. You can pair that with arithmetic coding to get a lossless compression/decompression algorithm. See https://en.wikipedia.org/wiki/Arithmetic_coding
In the way that you say, you can do lossless data compression, but then the LLM is used in a very distinct way than it is used in applications like chat or coding assistance. In the latter applications, you do queries which aim to extract information from the training data set, but which may return hallucinated content instead of correct content. If you use an LLM just to provide an estimation for the frequencies of…
Why would you use an LLM for that? The whole point is to encode contextual probabilities. So basically: given this prefix of text, what's are the probabilities for next tokens? You can use this conditional probability distribution to sample from to create plausible text, or you can use it for lossless compression. The math is very similar.
Earlier quoted context omitted.
Transformers do this for any stream of tokens, those tokens can map to anything you want, and you will get lossy compression. Text produced by humans just happens to be dense, available, and a useful prior, but it is not intrinsically required. See 3D vision transformers for example.
It is not possible to compress arbitrary data. If the data is already compressed, or it is encrypted, or it is randomly generated, it cannot be compressed with any method. This is foundational information theory. https://en.wikipedia.org/wiki/Lossless_compression#Limitatio...
Earlier quoted context omitted.
LLM compression doesn't necessarily have to be lossy. You can use the fact that LLMs predict P(next token | existing tokens) to losslessly and efficiently compress arbitrary token sequences. This idea is closely related to arithmetic coding.
True, but it's not relevant because that isn't how we actually train LLMs for use as quasi-intelligent tools. We specifically do not want the model to be able to just memorize its input, which is what your process requires. Many things about the process are similar, so there's some analogy, but it just isn't the same.