Compression is prediction
ngrok.com
Compression is prediction
1–10 of 324 posts
Re: Compression is prediction
#2I was rushing to post this and then found out somebody had already
Re: Compression is prediction
#3Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing.
[0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98
Re: Compression is prediction
#4[deleted]
Re: Compression is prediction
#5This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process!
Re: Compression is prediction
#6> compressors and LLMs
Why only LLMs? All statistical models are compressor. You can say "model" and "compressor" are synonyms.
Article does not mention "embeddings" at all, even though it's commonly viewed as a compression method. Also "encoder" part on "auto-encoders".
Re: Compression is prediction
#7Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing. [0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98
I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.
Re: Compression is prediction
#8This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process!
Incidentally, the relationship is bidirectional. You can try it out just for fun. zstd is a pretty crappy language model :)
Re: Compression is prediction
#9See also: Bellard's Lossless Data Compression With Neural Networks
Re: Compression is prediction
#10I was thinking about the same topic and the conclusion can be wrong. LLMs are compressors, but compressors are not LLMs. Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot.
Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some level of predictions. There are actually some papers that inject n-grams statistics as a part of the neural network weights.