Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

71–80 of 324 posts

Re: Compression is prediction

#71

Post by kazinator »

It's more or less obvious that the LLM is a lossy-compressed version of the training data; it reproduces sequences of tokens that are the sort of thing that could plausibly occur in the training data, and avoids sequences that are implausible.

Because most of the training data has good grammar, the LLM is strongly trained on grammar; it will rarely predict ungrammatical gibberish. Even if there are grammar mistakes in the data, they are not systematic and so don't reinforce each other.

Re: Compression is prediction

#72

Post by exe34 »

chermi wrote: post #48
farfatched wrote: post #36

This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University. > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine l…

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

Reminds me of Stephen Wolfram "discovering" things in the sense that other people would say "today I learnt".

Re: Compression is prediction

#73

Post by kazinator »

TeMPOraL wrote: post #62
chermi wrote: post #48

Earlier quoted context omitted.

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

And yet to this day, in AI threads, so many people act shocked and surprised if you dare follow the obvious implication and claim that understanding is a form of lossy compression.

Shorter description isn't understanding, let alone of it is lossy.

When you shorten a description in a lossy way, you are deciding a priori that some differences in the object don't matter, and it's not because you understand the object, but because it serves your goal of shortening the description.

Re: Compression is prediction

#74

Post by ablob »

TeMPOraL wrote: post #62
chermi wrote: post #48

Earlier quoted context omitted.

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

And yet to this day, in AI threads, so many people act shocked and surprised if you dare follow the obvious implication and claim that understanding is a form of lossy compression.

If you can predict other things by understanding something, is it truly lossy compression?

Re: Compression is prediction

#75

Post by SgtBastard »

jbay808 wrote: post #37
glial wrote: post #15

Earlier quoted context omitted.

> it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process This is not intuitive to me. It seems like a "new idea" is something that (almost by definition) isn't in the training set. Can you elaborate a bit? Edit: but perhaps a good model could arise from training, which would be a good idea in the sense that parsimonious ideas are good scientific ideas.

Imagine you're curve-fitting a bunch of data points on, say, the orbital motion of planets and asteroids. You get tons and tons of data on these orbital motions, and then put it into a huge black-box optimization algorithm that compresses the heck out of it. It compresess and compresses until it can't find a more compact representation, no matter how much more effort it applies. The output is a function, where you ca…

> A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data"

I just wanted to confirm your underlying point here: training a model isn’t about finding a function that fits the observed data (even though that’s the outcome) but instead finding an approximation of the unknown source that generated the source data in the first place.

In the case of LLMs an approximation not of a human but of the sum of humanity that produced the training copora?

Re: Compression is prediction

#76

Post by [deleted] »

ablob wrote: post #74
TeMPOraL wrote: post #62

Earlier quoted context omitted.

And yet to this day, in AI threads, so many people act shocked and surprised if you dare follow the obvious implication and claim that understanding is a form of lossy compression.

If you can predict other things by understanding something, is it truly lossy compression?

[deleted]

Re: Compression is prediction

#77

Post by pornel »

ssivark wrote: post #14

Nope; there is a bit more nuance and the distinction is important. Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems . The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support ! Eg: you observe a rare edge case in your training data and (lossy)…

You're just assuming a flawed implementation of a predictor and complain that it's flawed. That doesn't invalidate the general premise that predictors are compressors.

Predictors don't have to give you some corpus-average probability. They can give probabilities adjusted per symbol. The probability could even be exactly 1 for some symbol when the predictor is absolutely certain (and then it costs 0 bits to encode).

If you had a magic oracle predictor that already knew all the data you wanted to send, you'd have perfect compression.

Re: Compression is prediction

#78

Post by throw290483 »

I see it that prediction is a form of compression.

Say you have a computer file composed of two parts, the first represents the setup of an experiment, and the second is the data produced by the experiment.

If you have a good theory relating to this type of experiment, then you can predict much of the second part of the file. So you only need to store the first part and possibly some corrections to the least significant bits of some of the parts of the second part of the file.

Thus with good prediction, you can compress this type of file.

Re: Compression is prediction

#79

Post by altcognito »

ssivark wrote: post #14

Nope; there is a bit more nuance and the distinction is important. Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems . The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support ! Eg: you observe a rare edge case in your training data and (lossy)…

> Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems.

At some level, nothing is representative of all future "problems".

I think as far as corner cases is concerned, people can optimize for what they think matters. Larger models, higher sensitivities, better filtering.

Beyond this, I don't know what you're peeved by. There's no manifesto, there's no claiming that any particular method to prediction is perfect yet.

Re: Compression is prediction

#80

Post by mpalmer »

chermi wrote: post #48
farfatched wrote: post #36

This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University. > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine l…

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

I'm glad to see someone feels similarly. There is nothing wrong with ignorance, but there's no excuse mistaking learning for invention. Especially from someone bearing the title "Developer Educator"

I don't think it's the case here, but worth noting too that LLM-written blog posts adopt this tone seemingly by default.

Never the least bit of surprise, wonder, doubt, or frustration to get in the way of the steady staccato beat of metaphors, conclusions... and three-item lists.

Post reply on HN