31–40 of 324 posts
Nope; there is a bit more nuance and the distinction is important. Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems . The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support ! Eg: you observe a rare edge case in your training data and (lossy)…
Nope; there is a bit more nuance and the distinction is important. Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems . The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support ! Eg: you observe a rare edge case in your training data and (lossy)…
Also sparked the thought that the assumption only holds if the future looks like the present.
There is the Kolmogorov Complexity [1], Normalized Information Distance [2] and Normalized compression distance [3] that correlates those.
Finally, there's the Pre-Big Bang Informational Compression and the Delayed Release of Antimatter [4]
All big {rabbit/black} holes to lose some time, if you have any.
[0] https://en.wikipedia.org/wiki/Prediction_by_partial_matching
[1] https://en.wikipedia.org/wiki/Kolmogorov_complexity
[2] https://homepages.cwi.nl/~paulv/papers/chapter08.pdf
[3] https://en.wikipedia.org/wiki/Normalized_compression_distanc...
Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing. [0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98
I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.
> Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine learning still belong together. Brains are the ultimate compression and communication systems. And the state-of-the-art algorithms for both data compression and error-correcting codes use the same tools as machine learning.
Book (creative commons): https://www.inference.org.uk/mackay/itila/book.html
Lectures: https://m.youtube.com/playlist?list=PLruBu5BI5n4aFpG32iMbdWo...
This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process!
> it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process This is not intuitive to me. It seems like a "new idea" is something that (almost by definition) isn't in the training set. Can you elaborate a bit? Edit: but perhaps a good model could arise from training, which would be a good idea in the sense that parsimonious ideas are good scientific ideas.
Of course, one thing you get out of this is a great curve-fit for your existing data, which you can interpolate to find the position of any observed planet at any desired time.
But could this function also succeed in predicting the orbital motion of objects that aren't in the dataset? If I spot a new comet, and put it into the compressed function, would I get an accurate prediction of its orbital motion, even though that object wasn't in the training data?
The answer is "it depends, but probably yes". Newton's laws of orbital motion turn out to be simple compared to the size of the training data. So if the black-box compression has done a good job, it might output that function, or a close approximation of it. With a sufficient quantity of sufficiently accurate data, it might even improve on it; random errors can't be compressed, but where the deviations between observations and Newton's law turn out not to be random but rather the influence of an unobserved gravity source, or general relativity, the black-box algorithm will likely capture that as well.
A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data". Machine learning has proven to be quite good at solving inverse problems, and this is just a very abstract one of them.
(A forward-problem is something like solving for the electric fields from a set of charged particles; an inverse problem, https://en.wikipedia.org/wiki/Inverse_problem, is one where you have data on the electric fields at various points and want to reconstruct the arrangement of charged particles that produced it. Or more generally, you have sampled data on the output of an unknown process, and want to reconstruct the process that produced the data).
The inverse-problem-solving happens at the ML training step. The language model itself, that comes out of that, is solving the forward-problem: it has a generative-process baked in and now it's generating new data from it. But if the training process has done a good job of compression, it will certainly be able to generate valid new ideas that aren't in the training set, because the inverse model has solved for the underlying features of the real process that generated the training data, and those features can generate additional valid outputs that it wasn't trained on.
> compressors and LLMs Why only LLMs? All statistical models are compressor. You can say "model" and "compressor" are synonyms. Article does not mention "embeddings" at all, even though it's commonly viewed as a compression method. Also "encoder" part on "auto-encoders".
This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University. > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine l…