Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

181–190 of 324 posts

Re: Compression is prediction

#181

Post by ChuckMcM »

enneff wrote: post #174
ChuckMcM wrote: post #167

Earlier quoted context omitted.

I suspect you're being snarky :-) but this is a really interesting question, and one that has had a lot of research done. I'm not current (I stopped following folks doing this research closely around 2019) but what we 'didn't' know about how brains work was still huge. Signaling levels, enzymes, the connectome, quantum effects, it is a really deep question. That said, once we do get a working idea of how it works, an…

I think the point is that tremendous complexity can arise from relatively simple mechanisms. That is what life is, at many levels. I’m not at all convinced that the current LLM approach will yield something we can broadly call consciousness but saying that it’s a simple concept and therefore won’t support consciousness is a specious argument imo.

I completely agree, tremendous complexity can arise from simple mechanisms. Gleick's Chaos is a really good introduction to that. I was talking about the article though, and the mechanisms currently used for training and inference in LLMs. Those mechanisms are mathematically precise (unlike Chaotic attractors) and as the author points out, achieve the same function as compressors do in a strict bit pattern minimization role. Sometimes tensor math is pretty complex, like the FFT and DCTs on JPEG compression, but with the same inputs you get the same results. And while a JPEG will never decompress to a different image than the one that was compressed in the first place, LLMs do not 'infer' token streams that haven't been trained in their training process. The big difference here is that if you imagine a JPEG compressor that compresses 100 different images into one 'chunk', you can see how to provoke it to produce any one of the images it previously compressed. And with a bit of creativity you can have it express different images in different parts of the resulting composite. FWIW I looked at patenting something like this for digital cameras to give them more "shots" space for a given amount of SD storage.[1]

Given the way that models work in 'inference' mode (vs 'training' mode) you can't forward bias the result into the correct result when there are multiple forward results that have identical weights. It's the root cause of hallucinations, and you've lost information in the training phase that you can't then use to discriminate between the 'right' answer and an equally valid 'wrong' answer.

[1] FWIW I could never recover enough state to insure that the image it regenerated was all of the same image you took. So you might get the street but one of the houses might be a house that was in a different picture you took. That kind of bug. Mostly arising out of the same kind of problem you have with using hashes to find documents, when you get a hash collision two documents have the same hash, so you don't know which one to return.

Re: Compression is prediction

#183

Post by genxy »

beagle3 wrote: post #159
bonoboTP wrote: post #106

Earlier quoted context omitted.

Citing 2023 makes it seem like this is newer than it is. Compression, prediction and intelligence have long been known to be deeply connected.

You expect every blog post to find the earliest relevant paper to cite, just so one could look at the year (without reading said paper - which would have made clear that the connection isn’t recent) to assess novelty? I don’t think that’s reasonable. It’s a blog post. If it was, say, a peer reviewed paper by Hinton or LeCunn that fails to cite Schmidhuber, that would be reasonable criticism in my opinion. (Spoiler: t…

> You expect every blog post to find the earliest relevant paper to cite

This should be expected out of everyone. If you don't respect the reader enough to do this, why should we read your posts?

I think papers should be retracted for not citing prior art, even if you weren't aware of it.

Re: Compression is prediction

#184

Post by jjk166 »

The article is using probability where it really means proportion and prediction where it means evaluation. The mathematical equivalency is both much less surprising and less revealing once reframed.

If we consider the first example with the arithmetic code, the initial presupposition that only the characters A, B, and C appear in the string already reduces the entropy from 56 ascii bits to 14 bits (A vs Not A and B vs Not B for each character). If you further consider that you only need to distinguish B vs Not B if it's not A, then you can just represent As with a single zero bit and only represent the non-As as two bits (the first of which will necessarily always be a 1 bit). This gets you to 10 bits without even having the proportions of the string. Of course this would be a poor convention if there were say only a single A; in that worst case scenario you would need 13 bits, but simply knowing which character appears the most, without knowing by how much, 11 bits is the worst case scenario for a length 7 string with 3 potential characters. The last bit can be made implicit if you further choose the second conditional appropriately - i.e. if instead of B vs Not B we chose C vs Not C, our last bit would be zero and could simply be dropped meaning both 10 and a single 1 bit encode C - allowing you to encode the example string in just 9 bits and an arbitrary string of that length in 10, again regardless of proportions. That improvement over the arithmetic encoding result in the example is just a case of us cramming a little extra information into the encoding algorithm.

Arithmetic encoding is more clean and more easily extensible, it makes more sense to use than this custom encoding of 7 trits to binary but the point is the "probability" the article mentions is a superficial quality of life feature, not the secret sauce that is the actual key to compression.

Re: Compression is prediction

#185

Post by ballooney »

TacticalCoder wrote: post #104
smath wrote: post #94

Earlier quoted context omitted.

Ah Sir David MacKay. I so respect him. Great explainer and speaker. He had built this text entry tool called Dasher [0] - that I'd heard him introduce at Princeton around 2006. It was basically an early language model that predicted which characters are more likely than others, given what you've already types and it would adjust the sizes of the available next characters based on their probabilities. [0] https://dash…

> It was basically an early language model that predicted which characters are more likely than others, given what you've already types and it would adjust the sizes of the available next characters based on their probabilities. "adjust the sizes of the available next characters": you mean assigning less bits to characters the most likely to come next, and more bits to the least likely characters to come next? So it'…

It is infact a visual implementation of arithmetic coding, and he used it in his lectures to teach arithmetic coding in the lecture following the one on huffman coding. The video lectures posted earlier are highly recommended, best course I took as an undergrad

Re: Compression is prediction

#186

Post by qsera »

Dylan16807 wrote: post #175
qsera wrote: post #162

Earlier quoted context omitted.

I was speaking in the context of humans. When someone teaches you, the content coming from the teacher is very compressed. One decompress it when they can generalize and apply it. So understanding is compressed, but is not the act of compressing. I mean it is not compressed from a larger data or made by compressing a larger data. The larger data it represents never existed. It is like the definition of a fractal...

> I was speaking in the context of humans. So was I. > One decompress it when they can generalize and apply it. Generalizing is compression. Generalizing and applying are very different things. > So understanding is compressed, but is not the act of compressing. Understanding (noun) is compressed. Understanding (verb) is the act of creating understanding (noun) and is compression. > The larger data it represents neve…

>creating understanding (noun) and is compression.

What is it compressing from?

Re: Compression is prediction

#187

Post by zahlman »

bonoboTP wrote: post #106
farfatched wrote: post #96

Earlier quoted context omitted.

I don't think this is a fair critique. The author of the post uses standard terminology like entropy coding and arithmetic coding, and cited a paper "in 2023, Google DeepMind released a paper arguing that language modeling and compression are two views of the same thing" which discusses it further. This blog post is great. Well explained, and clearly took a lot of effort. I don't interpret it as them claiming to have…

Citing 2023 makes it seem like this is newer than it is. Compression, prediction and intelligence have long been known to be deeply connected.

That compression and prediction are deeply connected is obvious from simply considering how compression algorithms work.

Nothing about the concept of "intelligence" is nearly as well defined as the other two, nor as well agreed upon.

Re: Compression is prediction

#188

Post by pishpash »

Dylan16807 wrote: post #152
bonoboTP wrote: post #107

Earlier quoted context omitted.

I'm of two minds here. The pro is that the "you could have invented this" walkthrough from first principles is more engaging than "and then so and so introduced this term in 1972 and the definition is such and such". This style is a reaction to that boring and dry teaching style and tries to push towards what eg Feynman pointed at in the Brazil critique. The con is that you don't get to understand and see any of the…

There's no reason a compelling walkthrough can't throw in historical citations, is there?

Exactly. The best science exposition combines both. See, The Mechanical Universe and Veritasium.

Re: Compression is prediction

#189

Post by vrighter »

tgv wrote: post #178
Lerc wrote: post #138

Prediction is compression, but I am not sure if it is true the other way around. It's obvious that an accurate predictor enables encoding only the data that the predictor gets wrong. But a compressor can encode patterns that defy prediction by looking at the data as a whole. It doesn't have to look at everything in sequence as it arrives. Applying transformations prior to entropy encoding often isn't just 'rearrangin…

Indeed. If you're going for a catchy generalization, at least write it correctly. Most compression is history, and only extrapolates under the assumption that "nothing changes".

there are dictionary compressors (decent compression, most common, fast), and statistical compressors (better compression, slower). Statistical compressors are much closer to LLMs in that an llm is learning statistics about the data too.

And yes, compression is history, that's what statistics are all about. Statistics can only measure the past to make a prediction about the future. And LLMs work in the same way. The context is the history, and given that history, it predicts the next token. An LLM can, almost trivially, be dropped into something like the PPM statistical compressor (it's just replacing one implementation of a markov chain with another).

Re: Compression is prediction

#190

Post by pishpash »

Teever wrote: post #172
Dylan16807 wrote: post #152

Earlier quoted context omitted.

There's no reason a compelling walkthrough can't throw in historical citations, is there?

What specific historical citations would you like to see in this post? Perhaps you could contact the author with your suggestions?

They did not come up with the ideas themselves, so they got them somewhere. Follow the source and all the citations show up. It must be a modern thing where online blogging randos pretend they are all geniuses.
Post reply on HN