Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

171–180 of 324 posts

Re: Compression is prediction

#171

Post by vatsachak »

gr_norm wrote: post #44
porphyra wrote: post #42

Earlier quoted context omitted.

How does that invalidate the "compression is prediction"? If the future data is different and you failed to generalize, then the failure to predict means you got worse at compressing and have to spend more bits storing the new information. Conversely, if the future data is the same as that you've seen previously, you could predict it very well, and compress better as a result.

A maximally efficient compressor for the existing data distribution is not in general (and often will not be) maximally efficient for future data. The former may only be enabled by convenient local optima of the input distribution that a compressor accounting for the latter could not take advantage of. For instance, consider the distribution of strings drawn from the language '0+'. Now consider the same for the langu…

In a world where we see 0+ strings more often the first compressor would be better

Re: Compression is prediction

#172

Post by Teever »

Dylan16807 wrote: post #152
bonoboTP wrote: post #107

Earlier quoted context omitted.

I'm of two minds here. The pro is that the "you could have invented this" walkthrough from first principles is more engaging than "and then so and so introduced this term in 1972 and the definition is such and such". This style is a reaction to that boring and dry teaching style and tries to push towards what eg Feynman pointed at in the Brazil critique. The con is that you don't get to understand and see any of the…

There's no reason a compelling walkthrough can't throw in historical citations, is there?

What specific historical citations would you like to see in this post?

Perhaps you could contact the author with your suggestions?

Re: Compression is prediction

#173

Post by vatsachak »

cubefox wrote: post #165
davmre wrote: post #18

Earlier quoted context omitted.

Any compressor actually can be used, trivially, as an autoregressive language model. Given a context (for LLMs, this would include the entire pretraining dataset, plus the prompt), you compress `context + next_token` for every possible next token. The tokens that co-compress best with the existing context are the 'least surprising' continuations. Choose one of them and iterate. You can easily generate text with gzip…

The fact that the best compression algorithms are, in the LLM setting, still catastrophically bad at prediction suggests that the overlap between compression and prediction is weak at best.

https://bellard.org/ts_zip/

Re: Compression is prediction

#174

Post by enneff »

ChuckMcM wrote: post #167
sdenton4 wrote: post #163

Earlier quoted context omitted.

Let alone a sack of wet, self replicating protein! Just endless copying... How could it ever do anything more?

I suspect you're being snarky :-) but this is a really interesting question, and one that has had a lot of research done. I'm not current (I stopped following folks doing this research closely around 2019) but what we 'didn't' know about how brains work was still huge. Signaling levels, enzymes, the connectome, quantum effects, it is a really deep question. That said, once we do get a working idea of how it works, an…

I think the point is that tremendous complexity can arise from relatively simple mechanisms. That is what life is, at many levels. I’m not at all convinced that the current LLM approach will yield something we can broadly call consciousness but saying that it’s a simple concept and therefore won’t support consciousness is a specious argument imo.

Re: Compression is prediction

#175

Post by Dylan16807 »

qsera wrote: post #162
Dylan16807 wrote: post #153

Earlier quoted context omitted.

I would roughly say that understanding is compression and application is decompression. You don't memorize the specific details, you memorize the flow of systems. But you get back to the details to put things into practice.

I was speaking in the context of humans. When someone teaches you, the content coming from the teacher is very compressed. One decompress it when they can generalize and apply it. So understanding is compressed, but is not the act of compressing. I mean it is not compressed from a larger data or made by compressing a larger data. The larger data it represents never existed. It is like the definition of a fractal...

> I was speaking in the context of humans.

So was I.

> One decompress it when they can generalize and apply it.

Generalizing is compression. Generalizing and applying are very different things.

> So understanding is compressed, but is not the act of compressing.

Understanding (noun) is compressed. Understanding (verb) is the act of creating understanding (noun) and is compression.

> The larger data it represents never existed.

The larger data is everything you take in about the subject. Experience and instruction.

Re: Compression is prediction

#176

Post by Dylan16807 »

Teever wrote: post #172
Dylan16807 wrote: post #152

Earlier quoted context omitted.

There's no reason a compelling walkthrough can't throw in historical citations, is there?

What specific historical citations would you like to see in this post? Perhaps you could contact the author with your suggestions?

I'm speaking in general terms and I have not studied much on this subject so I don't have specific suggestions for this post. Please refer to earlier posts in this comment chain for a general idea of what citations would be useful.

Re: Compression is prediction

#177

Post by hnfong »

beagle3 wrote: post #159
bonoboTP wrote: post #106

Earlier quoted context omitted.

Citing 2023 makes it seem like this is newer than it is. Compression, prediction and intelligence have long been known to be deeply connected.

You expect every blog post to find the earliest relevant paper to cite, just so one could look at the year (without reading said paper - which would have made clear that the connection isn’t recent) to assess novelty? I don’t think that’s reasonable. It’s a blog post. If it was, say, a peer reviewed paper by Hinton or LeCunn that fails to cite Schmidhuber, that would be reasonable criticism in my opinion. (Spoiler: t…

Why would blog posts not be subject to such criticism?

Either the author knew of prior work that argues the same thing and they ignored it, or they didn't know. And if one writes a 1000+ word article premised on this idea, wouldn't one be presumed to know at least in which century the idea originated from?

Arguably these kind of blog posts should be more subject to such criticisms, because the blog posts purport to "teach" the general public about a concept in an authoritative tone (or at least the author seems to pose as knowledgeable in the subject), while for academic papers, everyone who actually reads the paper knows where the ideas came from anyway and it's mainly an issue of attribution (and maybe about fairly distributing the citation count...)

Re: Compression is prediction

#178

Post by tgv »

Lerc wrote: post #138

Prediction is compression, but I am not sure if it is true the other way around. It's obvious that an accurate predictor enables encoding only the data that the predictor gets wrong. But a compressor can encode patterns that defy prediction by looking at the data as a whole. It doesn't have to look at everything in sequence as it arrives. Applying transformations prior to entropy encoding often isn't just 'rearrangin…

Indeed. If you're going for a catchy generalization, at least write it correctly. Most compression is history, and only extrapolates under the assumption that "nothing changes".

Re: Compression is prediction

#179

Post by hnfong »

adrianN wrote: post #146
kazinator wrote: post #73

Earlier quoted context omitted.

Shorter description isn't understanding, let alone of it is lossy. When you shorten a description in a lossy way, you are deciding a priori that some differences in the object don't matter, and it's not because you understand the object, but because it serves your goal of shortening the description.

I'm not aware of a better definition of "understanding" that would allow me to tell whether some system "understands" some other system. Do you happen to know one?

See: A. M. Turing (1950) Computing Machinery and Intelligence. Mind 49: 433-460.

I mean, my interpretation is that the question Turing tried to answer is equivalent to "How can we determine whether machines understand humans/human thought?"

This only works when both systems can talk about pretty much arbitrary things, but if you want a more general method for less complex systems, perhaps having one system simulate another system is sufficient. (Which is also another Turing invention)

Re: Compression is prediction

#180

Post by vintermann »

Dylan16807 wrote: post #84
kazinator wrote: post #73

Earlier quoted context omitted.

Shorter description isn't understanding, let alone of it is lossy. When you shorten a description in a lossy way, you are deciding a priori that some differences in the object don't matter, and it's not because you understand the object, but because it serves your goal of shortening the description.

Or you actually do understand it. You can't just assume smaller is better but it often is. And very often it's more information-dense.

We model the data. The model, hopefully, captures something real in the data. If it does, then it's fair to say that we understand the data better.

But it's frankly a philosophical question what's real or not. No model is going to capture absolutely everything about the thing it models - at that point, it would be the thing. The best we can hope for is that it captures everything we care about.

And no experiment or metric can tell you if you care about the right things. At best it can tell us if we care about a thing given other things we care about. "No cares in, no cares out".

To make it a little more concrete: you could compress a string from back to front. You could build an LLM to help you do that. If you care about file size, that's almost certainly a bad idea, the forward LLM will be better for that purpose. But are there purposes for which the backward LLM might be better? I think that's not so hard to imagine. Often we wonder about "what came before".

Post reply on HN