Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

231–240 of 324 posts

Re: Compression is prediction

#231

Post by jll29 »

WithinReason wrote: post #209
bonoboTP wrote: post #106

Earlier quoted context omitted.

Citing 2023 makes it seem like this is newer than it is. Compression, prediction and intelligence have long been known to be deeply connected.

At least since the 1960s: https://en.wikipedia.org/wiki/Solomonoff%27s_theory_of_induc...

The two contributioms that come to mind are the Hutter cash prize for the best compressor of the English Wikipedia and the work on PPM compression-based text classification by the late Prof. Ian Witten's group at Waikato (NZ) [1,2]. The model that best compresses the input string was likely generated by the distribution from which the compression model was 'trained'

[1] https://news.ycombinator.com/item?id=27440093

[2] https://www.computer.org/csdl/proceedings-article/dcc/2000/0...

Re: Compression is prediction

#232

Post by FeepingCreature »

ChuckMcM wrote: post #181
enneff wrote: post #174

Earlier quoted context omitted.

I think the point is that tremendous complexity can arise from relatively simple mechanisms. That is what life is, at many levels. I’m not at all convinced that the current LLM approach will yield something we can broadly call consciousness but saying that it’s a simple concept and therefore won’t support consciousness is a specious argument imo.

I completely agree, tremendous complexity can arise from simple mechanisms. Gleick's Chaos is a really good introduction to that. I was talking about the article though, and the mechanisms currently used for training and inference in LLMs. Those mechanisms are mathematically precise (unlike Chaotic attractors) and as the author points out, achieve the same function as compressors do in a strict bit pattern minimizati…

I do not understand this intuition that "true consciousness has to be random". The things that make me me are highly deterministic!

> LLMs do not 'infer' token streams that haven't been trained in their training process

While we're at it, this is simply untrue (in-context learning) unless you generalize "token streams" so radically that it could be readily analogized to humans as well.

Re: Compression is prediction

#233

Post by bonoboTP »

zahlman wrote: post #187
bonoboTP wrote: post #106

Earlier quoted context omitted.

Citing 2023 makes it seem like this is newer than it is. Compression, prediction and intelligence have long been known to be deeply connected.

That compression and prediction are deeply connected is obvious from simply considering how compression algorithms work. Nothing about the concept of "intelligence" is nearly as well defined as the other two, nor as well agreed upon.

"Understanding" is clearly linked to compression. Taking observations and coming up with a more compact representation that explains them, analogous to coming up with a compact set of axioms that generate facts, or a small Turing machine or short program that generates a list of strings.

Intelligence is a broader concept but definitely involves understanding how a system/envoronment works and making predictions about its unfolding, especially actionable ones that allow you to steer that state towards some goal states.

These are not new.

Re: Compression is prediction

#234

Post by bonoboTP »

teekert wrote: post #116
chermi wrote: post #48

Earlier quoted context omitted.

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

The post says this is all part of gzip and LLMs, what are you saying? I’ve been using gzip my entire life. I read between the lines “this is common knowledge” throughout the piece. Throwing in some names and dates only makes this super clear story harder to read (and more like studying then the playful exploration this post was intended as).

A short paragraph at the end on the origin of these ideas can be an easy way to dispel the misconception of potential beginner readers that the insights are novel.

Re: Compression is prediction

#235

Post by hnfong »

adrianN wrote: post #208
hnfong wrote: post #179

Earlier quoted context omitted.

See: A. M. Turing (1950) Computing Machinery and Intelligence. Mind 49: 433-460. I mean, my interpretation is that the question Turing tried to answer is equivalent to "How can we determine whether machines understand humans/human thought?" This only works when both systems can talk about pretty much arbitrary things, but if you want a more general method for less complex systems, perhaps having one system simulate a…

There are many people who would claim that passing the Turing test is insufficient to show "understanding" (compare for example the Chinese Room thought experiment).

Yes but it's (kind of?) a definition as you asked for.

At this point, I am unaware of a better definition. I know the Chinese Room argument (and I disagree with it), but I'm not aware whether the proponents of that argument have a better definition of understanding other than "well, the Turing Test isn't enough"...

---

PS: Interestingly the issue of compression is highly relevant regarding the Chinese Room argument -- the essential element in the Chinese Room argument is that the information is not compressed...

Re: Compression is prediction

#236

Post by tgv »

vrighter wrote: post #189
tgv wrote: post #178

Earlier quoted context omitted.

Indeed. If you're going for a catchy generalization, at least write it correctly. Most compression is history, and only extrapolates under the assumption that "nothing changes".

there are dictionary compressors (decent compression, most common, fast), and statistical compressors (better compression, slower). Statistical compressors are much closer to LLMs in that an llm is learning statistics about the data too. And yes, compression is history, that's what statistics are all about. Statistics can only measure the past to make a prediction about the future. And LLMs work in the same way. The…

Anything can only represent past measurements. Statistics is not an exception. But they don't make a prediction about the future. That comes from a model you have, and it often is implicit: "the linear trend from the last 12 months will hold in the next month" or whatever. So compression isn't by definition prediction. The other way around doesn't have to hold either, but in the case of LLMs it does.

Re: Compression is prediction

#237

Post by dannyw »

tptacek wrote: post #100
QuadrupleA wrote: post #67

Ted Chiang made a similar point in his article "ChatGPT is a blurry JPEG of the web" a few years ago: https://www.newyorker.com/tech/annals-of-technology/chatgpt-...

It's a great line, but that's obviously not all it is. You don't get new results in e.g. mathematics by looking carefully at the pixels of a JPEG.

If you have a validator for if a JPEG is a valid proof in mathematics, and can generate lots of "plausible" JPEGs, then you can arrive at hew results in mathematics.

Re: Compression is prediction

#239

Post by detourdog »

rcxdude wrote: post #229
Dylan16807 wrote: post #152

Earlier quoted context omitted.

There's no reason a compelling walkthrough can't throw in historical citations, is there?

Often the most straightforward way to walk through an idea while teaching it is not the same order that the ideas were developed, and might not even use the same set of ideas in building up to it, so it can be tricky to get both the best explanation of the idea and the historical context in at the same time without making things more confusing.

I like to see ideas presented as the evolved. Each solution is developed as a perceived reaction to the shortcomings of the previous. This becomes a contrast and comparison as to why one idea is appropriate for a particular context.

Re: Compression is prediction

#240

Post by ziofill »

Compression and error correction also go hand in hand: in compressed data every bit carries more information and therefore errors are more detrimental. This is one of the results that Shannon phrased exactly in terms of entropy.

My PhD supervisor had a beautiful example. Take the English message "errors can make messages unreadable" and ‘compress it’ by removing vowels. It’s still intelligible because English has redundancy:

rrrs cn mk mssgs nrdbl

Similarly, an uncompressed message with errors (swapped characters) is also intelligible because of the redundancy of English:

erwurs lan nake wesaagis unfeatable

But now we do both: we compress the message AND add errors. The result should be much harder (if not impossible) to read:

rwrs ln nk wssgs nftbl

Post reply on HN