Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

311–320 of 324 posts

Re: Compression is prediction

#311

Post by vrighter »

qsera wrote: post #162
Dylan16807 wrote: post #153

Earlier quoted context omitted.

I would roughly say that understanding is compression and application is decompression. You don't memorize the specific details, you memorize the flow of systems. But you get back to the details to put things into practice.

I was speaking in the context of humans. When someone teaches you, the content coming from the teacher is very compressed. One decompress it when they can generalize and apply it. So understanding is compressed, but is not the act of compressing. I mean it is not compressed from a larger data or made by compressing a larger data. The larger data it represents never existed. It is like the definition of a fractal...

I think of it as more "providing you with the model you can use to decompress". You only really learn something when you do it yourself, not just listen to someone talk about it.

Re: Compression is prediction

#312

Post by vrighter »

jbay808 wrote: post #129
SgtBastard wrote: post #75

Earlier quoted context omitted.

> A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data" I just wanted to confirm your underlying point here: training a model isn’t about finding a function that fits the observed data (even though that’s the outcome) but instead finding an ap…

Yes, exactly. And it's the compression that makes this happen. If it was just curve-fitting then the LLM really would act like a stochastic parrot, and it wouldn't generalize; also the model would also need to be much much bigger. As it is, the models are very big but they're still tiny compared to the dataset that they compress, so the compression process forces them to approximate a solution to the inverse problem.…

except that it's really not. It's about finding a function to fit the data. And "generalizing" is only when the question is jsut barely outside their training data. Like a truncated taylor series, is still somewhat accurate just outside the region it was designed for, But I couldn't say a truncated taylor series generalizes over the function it's approximating.

Re: Compression is prediction

#313

Post by krackers »

andai wrote: post #309
krackers wrote: post #288

Earlier quoted context omitted.

>I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric This is the premise of https://huggingface.co/spaces/Jellyfish042/UncheatableEval

Nice. This ranking basically matches other benchmarks, from what I can tell. Which implies this would probably also hold for the larger models, which are sadly not included in the leaderboard.

I think it's limited to using base models (non post-trained), because the post-training would skew the logit distribution. There are ways to "coax" post-trained models back into behaving "like" a base model, I wonder if the benchmark could be unofficially updated with those somehow.

Re: Compression is prediction

#314

Post by versteegen »

vrighter wrote: post #312
jbay808 wrote: post #129

Earlier quoted context omitted.

Yes, exactly. And it's the compression that makes this happen. If it was just curve-fitting then the LLM really would act like a stochastic parrot, and it wouldn't generalize; also the model would also need to be much much bigger. As it is, the models are very big but they're still tiny compared to the dataset that they compress, so the compression process forces them to approximate a solution to the inverse problem.…

except that it's really not. It's about finding a function to fit the data. And "generalizing" is only when the question is jsut barely outside their training data. Like a truncated taylor series, is still somewhat accurate just outside the region it was designed for, But I couldn't say a truncated taylor series generalizes over the function it's approximating.

I urge you to reconsider your beliefs. You are missing something important because you are thinking in terms of low-dimensional statistics. Deep learning doesn't just fit data, it finds features (abstractions) of the data.

Re: Compression is prediction

#315

Post by vrighter »

versteegen wrote: post #314
vrighter wrote: post #312

Earlier quoted context omitted.

except that it's really not. It's about finding a function to fit the data. And "generalizing" is only when the question is jsut barely outside their training data. Like a truncated taylor series, is still somewhat accurate just outside the region it was designed for, But I couldn't say a truncated taylor series generalizes over the function it's approximating.

I urge you to reconsider your beliefs. You are missing something important because you are thinking in terms of low-dimensional statistics. Deep learning doesn't just fit data, it finds features (abstractions) of the data.

this is not a belief. it is a fact. Doing the same thing in a higher dimensional space is still doing the same thing.

Re: Compression is prediction

#316

Post by mainmailman »

detourdog wrote: post #260
mainmailman wrote: post #251

Earlier quoted context omitted.

Have any examples of a blog post that meets your standard?

They are not blog posts but it is a presentation of technical development that meets my standard. james burke connections https://www.youtube.com/watch?v=XetplHcM7aQ&pp=ygUXamFtZXMgY... Tim Hunkin Secret Life of Machines https://www.youtube.com/watch?v=CJlrbMHLBd4&list=PLtaR0lZhSy... I have read plenty of blog posts that meet my standard. If I can think of one I will post it.

Thanks for the recommendations!

Re: Compression is prediction

#317

Post by SgtBastard »

vrighter wrote: post #315
versteegen wrote: post #314

Earlier quoted context omitted.

I urge you to reconsider your beliefs. You are missing something important because you are thinking in terms of low-dimensional statistics. Deep learning doesn't just fit data, it finds features (abstractions) of the data.

this is not a belief. it is a fact. Doing the same thing in a higher dimensional space is still doing the same thing.

I asked the original commenter to confirm my understanding of what they were saying.

I find it a fascinating alternative view to what is largely well understood (your counter-point).

The thing that stood out is the comment that it’s an approximation of the original data generator (humanity). Early approximations were poor (GPT 2-3, to an extent GPT-4).

I’m not so sure I can reject the hypothesis that such an approximation can be found.

Re: Compression is prediction

#318

Post by JohnKemeny »

adamgordonbell wrote: post #16

Small world. I just did a podcast on this same topic, but coming at it from a different direction, ie. me and my neighbor trying to beat the hutter prize for compression. Hutter Prize being where you are paid if you can compress wikipedia small enough. LLMs do very well at that, if, big if, you ignore the cost of initial weights. A cool Claude Shannon story: Shannon wanted to measure how much information is actually…

> ignore the cost of initial weights

Well, then Wikipedia itself is a very good compression that only needs the title to perfectly predict the full article.

Re: Compression is prediction

#319

Post by adamgordonbell »

JohnKemeny wrote: post #318
adamgordonbell wrote: post #16

Small world. I just did a podcast on this same topic, but coming at it from a different direction, ie. me and my neighbor trying to beat the hutter prize for compression. Hutter Prize being where you are paid if you can compress wikipedia small enough. LLMs do very well at that, if, big if, you ignore the cost of initial weights. A cool Claude Shannon story: Shannon wanted to measure how much information is actually…

> ignore the cost of initial weights Well, then Wikipedia itself is a very good compression that only needs the title to perfectly predict the full article.

Except the LLM generalizes its encoding to all english text where as the copy of wikipedia can only 'compress' wikipedia.

Re: Compression is prediction

#320

Post by JohnKemeny »

adamgordonbell wrote: post #319
JohnKemeny wrote: post #318

Earlier quoted context omitted.

> ignore the cost of initial weights Well, then Wikipedia itself is a very good compression that only needs the title to perfectly predict the full article.

Except the LLM generalizes its encoding to all english text where as the copy of wikipedia can only 'compress' wikipedia.

I probably didn't understand what you meant by

> Hutter Prize being where you are paid if you can compress wikipedia small enough. LLMs do very well at that, if, big if, you ignore the cost of initial weights.

then.

If all you care about is compressing Wikipedia, but ignore the size of the actual data, what is it that you are actually trying to do?

Post reply on HN