Compression is prediction

Post by larodi » Wed, Aug 12, 2026, 7:15 AM UTC

epistasis wrote: post #57

zahrevsky wrote: post #7

Earlier quoted context omitted.

I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.

Anybody working in the field will be very familiar with these concepts.

Post by WithinReason » Wed, Aug 12, 2026, 7:20 AM UTC

qsera wrote: post #186

Dylan16807 wrote: post #175

Earlier quoted context omitted.

> I was speaking in the context of humans. So was I. > One decompress it when they can generalize and apply it. Generalizing is compression. Generalizing and applying are very different things. > So understanding is compressed, but is not the act of compressing. Understanding (noun) is compressed. Understanding (verb) is the act of creating understanding (noun) and is compression. > The larger data it represents neve…

>creating understanding (noun) and is compression. What is it compressing from?

Post by jwr » Wed, Aug 12, 2026, 7:22 AM UTC

chermi wrote: post #48

farfatched wrote: post #36

This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University. > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine l…

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

Post by kuboble » Wed, Aug 12, 2026, 7:29 AM UTC

hnfong wrote: post #177

beagle3 wrote: post #159

Earlier quoted context omitted.

You expect every blog post to find the earliest relevant paper to cite, just so one could look at the year (without reading said paper - which would have made clear that the connection isn’t recent) to assess novelty? I don’t think that’s reasonable. It’s a blog post. If it was, say, a peer reviewed paper by Hinton or LeCunn that fails to cite Schmidhuber, that would be reasonable criticism in my opinion. (Spoiler: t…

Why would blog posts not be subject to such criticism? Either the author knew of prior work that argues the same thing and they ignored it, or they didn't know. And if one writes a 1000+ word article premised on this idea, wouldn't one be presumed to know at least in which century the idea originated from? Arguably these kind of blog posts should be more subject to such criticisms, because the blog posts purport to "…

Post by Dylan16807 » Wed, Aug 12, 2026, 7:41 AM UTC

vintermann wrote: post #180

Dylan16807 wrote: post #84

Earlier quoted context omitted.

Or you actually do understand it. You can't just assume smaller is better but it often is. And very often it's more information-dense.

We model the data. The model, hopefully, captures something real in the data. If it does, then it's fair to say that we understand the data better. But it's frankly a philosophical question what's real or not. No model is going to capture absolutely everything about the thing it models - at that point, it would be the thing. The best we can hope for is that it captures everything we care about. And no experiment or m…