101–110 of 324 posts
Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing. [0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98
I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.
Nope; there is a bit more nuance and the distinction is important. Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems . The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support ! Eg: you observe a rare edge case in your training data and (lossy)…
You're just assuming a flawed implementation of a predictor and complain that it's flawed. That doesn't invalidate the general premise that predictors are compressors. Predictors don't have to give you some corpus-average probability. They can give probabilities adjusted per symbol. The probability could even be exactly 1 for some symbol when the predictor is absolutely certain (and then it costs 0 bits to encode). I…
How about dictionary based compression as a counter example? Or the zig zag encoding scheme used in JPEG? I find it difficult to cast some of the things that effectively compress data as prediction.
That corresponds to PCFG models.
This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University. > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine l…
Ah Sir David MacKay. I so respect him. Great explainer and speaker. He had built this text entry tool called Dasher [0] - that I'd heard him introduce at Princeton around 2006. It was basically an early language model that predicted which characters are more likely than others, given what you've already types and it would adjust the sizes of the available next characters based on their probabilities. [0] https://dash…
"adjust the sizes of the available next characters": you mean assigning less bits to characters the most likely to come next, and more bits to the least likely characters to come next?
So it's not totally unlike adaptive Huffman coding but instead of updating the tree on already visited characters, here the tree with the weights of the possible following character is predictive?
Many of the debates in the comments seem to come down to whether people believe prediction and extrapolation are synonymous.
Earlier quoted context omitted.
I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…
I don't think this is a fair critique. The author of the post uses standard terminology like entropy coding and arithmetic coding, and cited a paper "in 2023, Google DeepMind released a paper arguing that language modeling and compression are two views of the same thing" which discusses it further. This blog post is great. Well explained, and clearly took a lot of effort. I don't interpret it as them claiming to have…
This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University. > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine l…
I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…
The con is that you don't get to understand and see any of the history of the ideas or even the ballpark when it was discovered, you attribute it to the blog mentally and you don't know what is how new or old and can't reference it properly when talking to others.
Earlier quoted context omitted.
Ah Sir David MacKay. I so respect him. Great explainer and speaker. He had built this text entry tool called Dasher [0] - that I'd heard him introduce at Princeton around 2006. It was basically an early language model that predicted which characters are more likely than others, given what you've already types and it would adjust the sizes of the available next characters based on their probabilities. [0] https://dash…
> It was basically an early language model that predicted which characters are more likely than others, given what you've already types and it would adjust the sizes of the available next characters based on their probabilities. "adjust the sizes of the available next characters": you mean assigning less bits to characters the most likely to come next, and more bits to the least likely characters to come next? So it'…
Try it out: `apt install dasher`. https://packages.debian.org/sid/dasher
Earlier quoted context omitted.
I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…
And yet to this day, in AI threads, so many people act shocked and surprised if you dare follow the obvious implication and claim that understanding is a form of lossy compression.
did the SSL cert expire? i'm getting a big scary warning about this blog