Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

161–170 of 322 posts

Re: Compression is prediction

#161

Post by jdthedisciple »

chermi wrote: post #48
farfatched wrote: post #36

This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University. > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine l…

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

I'm glad you're pointing this out because not only are these old insights, but I'm also pretty sure I've seen variations of this blog post years ago on even HN already.

The author acting as if they discovered this independently had me feel the exact same way. Kinda irritating and almost ... disrespectful? Not sure of the right words to describe it tbh

Re: Compression is prediction

#162

Post by qsera »

Dylan16807 wrote: post #153
qsera wrote: post #148

Earlier quoted context omitted.

>understanding is a form of lossy compression.. No. Teaching is compression. Understanding and application is decompression.

I would roughly say that understanding is compression and application is decompression. You don't memorize the specific details, you memorize the flow of systems. But you get back to the details to put things into practice.

I was speaking in the context of humans. When someone teaches you, the content coming from the teacher is very compressed. One decompress it when they can generalize and apply it. So understanding is compressed, but is not the act of compressing. I mean it is not compressed from a larger data or made by compressing a larger data. The larger data it represents never existed. It is like the definition of a fractal...

Re: Compression is prediction

#163

Post by sdenton4 »

ChuckMcM wrote: post #160
farfatched wrote: post #36

This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University. > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine l…

It also helps explain to people that LLMs are as likely as bzip to develop "consciousness".

Let alone a sack of wet, self replicating protein! Just endless copying... How could it ever do anything more?

Re: Compression is prediction

#165

Post by cubefox »

davmre wrote: post #18
pjankiewicz wrote: post #10

I was thinking about the same topic and the conclusion can be wrong. LLMs are compressors, but compressors are not LLMs. Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot. Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some lev…

Any compressor actually can be used, trivially, as an autoregressive language model. Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate. You can easily generate text with gzip…

The fact that the best compression algorithms are, in the LLM setting, still catastrophically bad at prediction suggests that the overlap between compression and prediction is weak at best.

Re: Compression is prediction

#166

Post by jdthedisciple »

There is a correct sense, but we're sort of garbling concepts here:

Predictability is the inverse of information density.

Low information density enables high compression, and vice versa.

It's called entropy. This is basic information theory to be quite frank..

Re: Compression is prediction

#167

Post by ChuckMcM »

sdenton4 wrote: post #163
ChuckMcM wrote: post #160

Earlier quoted context omitted.

It also helps explain to people that LLMs are as likely as bzip to develop "consciousness".

Let alone a sack of wet, self replicating protein! Just endless copying... How could it ever do anything more?

I suspect you're being snarky :-) but this is a really interesting question, and one that has had a lot of research done. I'm not current (I stopped following folks doing this research closely around 2019) but what we 'didn't' know about how brains work was still huge. Signaling levels, enzymes, the connectome, quantum effects, it is a really deep question.

That said, once we do get a working idea of how it works, and can perhaps synthesize a brain artificially with proteins, it will inform us on the next steps for silicon realization of that.

Re: Compression is prediction

#168

Post by vasco »

jdthedisciple wrote: post #161
chermi wrote: post #48

Earlier quoted context omitted.

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

I'm glad you're pointing this out because not only are these old insights, but I'm also pretty sure I've seen variations of this blog post years ago on even HN already. The author acting as if they discovered this independently had me feel the exact same way. Kinda irritating and almost ... disrespectful? Not sure of the right words to describe it tbh

Fabrice Bellard published this in 2023: https://bellard.org/ts_zip/

> The ts_zip utility can compress (and hopefully decompress) text files using a Large Language Model. The compression ratio is much higher than with other compression tools.

It's not only an old idea it's been totally done already.

Re: Compression is prediction

#170

Post by ssivark »

variadix wrote: post #59
ssivark wrote: post #14

Nope; there is a bit more nuance and the distinction is important. Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems . The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support ! Eg: you observe a rare edge case in your training data and (lossy)…

I’m not exactly sure what distinction you are trying to make, but a compressor that generalizes from currently observed day to potential future data _is_ a better compressor, as it will have better predictions as it observes more of the source data distribution. Maybe you are making a point about whether compressors can “learn” after compressing the input data, but this also isn’t a fundamental property of compressor…

But your definition of compression as prediction is tautological. How would you operationalize that into a measurable metric of compression?

The crucial idea is that once you do that and start optimizing for compression, at some point more compression on the training dataset can lead to worse predictions on the futures you care for. Which is very incompatible with the poster statement.

Post reply on HN