Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

81–90 of 324 posts

Re: Compression is prediction

#81

Post by pornel »

throw290483 wrote: post #78

I see it that prediction is a form of compression. Say you have a computer file composed of two parts, the first represents the setup of an experiment, and the second is the data produced by the experiment. If you have a good theory relating to this type of experiment, then you can predict much of the second part of the file. So you only need to store the first part and possibly some corrections to the least signific…

Another example is encrypted data. Statistically, encrypted data is indistinguishable from random. Truly random data is impossible to compress losslessly.

But if you had a predictor so smart that it could crack the encryption key, it could start predicting the rest of the encrypted stream, and therefore compress it.

Re: Compression is prediction

#82

Post by bch »

mpalmer wrote: post #80
chermi wrote: post #48

Earlier quoted context omitted.

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

I'm glad to see someone feels similarly. There is nothing wrong with ignorance, but there's no excuse mistaking learning for invention. Especially from someone bearing the title "Developer Educator" I don't think it's the case here, but worth noting too that LLM-written blog posts adopt this tone seemingly by default. Never the least bit of surprise, wonder, doubt, or frustration to get in the way of the steady stacc…

> I'm glad to see someone feels similarly. There is nothing wrong with ignorance, but there's no excuse mistaking learning for invention. Especially from someone bearing the title "Developer Educator"

>> a Developer Educator at ngrok with a passion for nerd-sniping developers.

Maybe more the latter than former...

Re: Compression is prediction

#83

Post by jagged-chisel »

exe34 wrote: post #72
chermi wrote: post #48

Earlier quoted context omitted.

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

Reminds me of Stephen Wolfram "discovering" things in the sense that other people would say "today I learnt".

I find it bothersome that language works this way. You can spend your whole life discovering things that are well known by the rest of the world. But the minute that you mention to a large group that you “discovered” it, suddenly you’re taking credit for discovering it for all of mankind.

Re: Compression is prediction

#84

Post by Dylan16807 »

kazinator wrote: post #73
TeMPOraL wrote: post #62

Earlier quoted context omitted.

And yet to this day, in AI threads, so many people act shocked and surprised if you dare follow the obvious implication and claim that understanding is a form of lossy compression.

Shorter description isn't understanding, let alone of it is lossy. When you shorten a description in a lossy way, you are deciding a priori that some differences in the object don't matter, and it's not because you understand the object, but because it serves your goal of shortening the description.

Or you actually do understand it.

You can't just assume smaller is better but it often is. And very often it's more information-dense.

Re: Compression is prediction

#85

Post by pornel »

bob1029 wrote: post #68

How about dictionary based compression as a counter example? Or the zig zag encoding scheme used in JPEG? I find it difficult to cast some of the things that effectively compress data as prediction.

Dictionary-based compression is based on prediction that recently seen words will be used again. That happens to be generally true for lots of datasets, including human languages (zipf distribution).

JPEG's zig-zag is a primitive for quantization, throwing data away based on rough approximation of human perception and biology. That isn't compression itself. However, the rounded and zeroed-out data is then compressed using a combination of RLE and Huffman, set up to predict the data will have lots of zeroes and few other distinct values (which the earlier step forces to be true). Or if you think about the system as a whole, you could say that JPEG predicts images will be blocky low-frequency patterns of DCT.

Re: Compression is prediction

#87

Post by chermi »

exe34 wrote: post #72
chermi wrote: post #48

Earlier quoted context omitted.

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

Reminds me of Stephen Wolfram "discovering" things in the sense that other people would say "today I learnt".

Wolfram was my example in my original rant!

Re: Compression is prediction

#88

Post by iwontberude »

AlotOfReading wrote: post #60
gr_norm wrote: post #44

Earlier quoted context omitted.

A maximally efficient compressor for the existing data distribution is not in general (and often will not be) maximally efficient for future data. The former may only be enabled by convenient local optima of the input distribution that a compressor accounting for the latter could not take advantage of. For instance, consider the distribution of strings drawn from the language '0+'. Now consider the same for the langu…

It might not be optimal, but it's not wrong to call it the best available guess. That's basically assuming Occam's razor / Solomonoff induction. Hutter published a bunch of work about what it means to have an "optimal" compressor and famously spent the past couple decades running a compression contest on the idea that it'd lead to insights in AI.

[dead]

Re: Compression is prediction

#89

Post by omneity »

ssivark wrote: post #14

Nope; there is a bit more nuance and the distinction is important. Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems . The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support ! Eg: you observe a rare edge case in your training data and (lossy)…

I'd approach this distinction differently. Prediction from compression is valid within the distribution of the compressed data. Which brings it much closer to LLMs in this case (can an LLM talk about a topic it has never seen in training? unlikely if it cannot be derived from other training data)

Re: Compression is prediction

#90

Post by [deleted] »

jagged-chisel wrote: post #83
exe34 wrote: post #72

Earlier quoted context omitted.

Reminds me of Stephen Wolfram "discovering" things in the sense that other people would say "today I learnt".

I find it bothersome that language works this way. You can spend your whole life discovering things that are well known by the rest of the world. But the minute that you mention to a large group that you “discovered” it, suddenly you’re taking credit for discovering it for all of mankind.

[deleted]
Post reply on HN