Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

121–130 of 324 posts

Re: Compression is prediction

#121

Post by AnotherGoodName »

ssivark wrote: post #14

Nope; there is a bit more nuance and the distinction is important. Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems . The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support ! Eg: you observe a rare edge case in your training data and (lossy)…

The record setting file compressors are all very generalizable. This comes about because you score compression by adding the program size (including any training data or dictionary) to the compressed data. If they didn't score it this way you could just ship a dictionary of the test data and your program just prints that. So of course they combine it and effectively measure the Kolmorogov complexity.

This means that training data isn't that helpful to top of the line compressors. eg. Fabrice Bellards nncp that's currently the record holding compressor for enwik9 is an LLM that learns on the fly. It's 628KB as shipped (LLM code complexity is high) and it wins on data sets gigabytes in size due to it's ability to create the training data on the fly. No pre-trained LLM comes close when you measure it as above (program size + compressed result).

So it's really not worth that much concern. There's a reason we all say AI is compression and we dismiss comments like the above. They don't pan out. No one's shipping significant dictionaries or pre-trained data. You want to win competitive compression? You'd better have raw code that learns on the fly and is naturally generalizable.

Re: Compression is prediction

#122

Post by versteegen »

nonameiguess wrote: post #45
throwaway_7274 wrote: post #5

This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process!

There's another element to this that I almost never see discussed. Ideas are not facts. Neither LLMs nor humans can generate new knowledge , as opposed to ideas, by thinking alone. Physical investigation and experimentation is necessary. The exception being pure mathematics since it exists solely in the realm of ideas. I'm willing to call that knowledge, but it's still a distinction, the old analytic/synthetic dichot…

That conception of knowledge is interesting, but I think using the label 'knowledge' for it is very problematic, it's too far from common definitions. The fact that you have to carve out an exception for mathematics already shows there's a problem. Because if maths, shouldn't thought experiments also produce new knowledge? You're excluding special and general relativity. It seems to me that what the concept actually describes is "information about the world".

Re: Compression is prediction

#124

Post by AlotOfReading »

versteegen wrote: post #117
AlotOfReading wrote: post #60

Earlier quoted context omitted.

It might not be optimal, but it's not wrong to call it the best available guess. That's basically assuming Occam's razor / Solomonoff induction. Hutter published a bunch of work about what it means to have an "optimal" compressor and famously spent the past couple decades running a compression contest on the idea that it'd lead to insights in AI.

There is a distinction between a compressor for a fixed dataset and one for an unknown population from which we have a sample. The optimal compressor for the sample may be the single best guess for the population, but that's not what Solomonoff induction does. It begins with a prior that allows all possible programs, and it never assigns all probability to the single optimal compressor, so it has no problem with the…

I think it's better to think of the hutter prize as a challenge to see how close people can get to efficiently approximating AIXI without the exponential cost of AIXItl. The fact that winners don't generalize well is just that they're not great approximations, because the behavior is different in the limit.

Re: Compression is prediction

#126

Post by versteegen »

sgsjchs wrote: post #91
duskwuff wrote: post #26

Earlier quoted context omitted.

A common design in compressors is to use LZ as a first step, but to then represent the constant data and/or offset-length pairs from LZ using an entropy coder. Deflate (as used in gzip) uses a Huffman coder. LZMA (as used by xz) uses a predictive range coder. Zstandard can use either Huffman or FSE. Some high-speed compressors like LZ4 skip the entropy coding stage entirely at the expense of compression ratio. Bzip2…

The first LZ-step pretty much directly maps to BPE tokenization in LLMs.

If doesn't correspond cleanly. I can see why you draw the link, because LZ compression will replace words with symbols but BPE is a non-contextual entropy encoding while LZ is contextual and adaptive and that makes it very different. I think BPE actually has more in common with Huffman encoding.

Re: Compression is prediction

#127

Post by AnotherGoodName »

throwaway_7274 wrote: post #5

This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process!

Prediction is literally what's allowed computers to make amazingly creative chess and go moves that a human would never have thought of.

More generally prediction allows you to path find towards a solution to reach some goal that no human might ever see.

If someone states "LLMs can't have new ideas because they only predict" you have to laugh. Prediction literally enables new ideas as you use those predictions to path find to a goal.

Re: Compression is prediction

#128

Post by walrus01 »

On a slightly related topic, static on disk files of LLMs are not incompressible, I have a number of "archived, maybe I'll use it later" Q8 quantized GGUF files that are about 90% of their original file size when run through xz with default options. It's not a ton of disk space savings, but disk space also isn't as cheap as it used to be. BF16 GGUFs will compress a lot.

Re: Compression is prediction

#129

Post by jbay808 »

SgtBastard wrote: post #75
jbay808 wrote: post #37

Earlier quoted context omitted.

Imagine you're curve-fitting a bunch of data points on, say, the orbital motion of planets and asteroids. You get tons and tons of data on these orbital motions, and then put it into a huge black-box optimization algorithm that compresses the heck out of it. It compresess and compresses until it can't find a more compact representation, no matter how much more effort it applies. The output is a function, where you ca…

> A lot of people seem to think of the training process as curve-fitting data (the "stochastic parrot" model), but I think of it more as "solving an inverse problem to approximate the unknown source that generated the training data" I just wanted to confirm your underlying point here: training a model isn’t about finding a function that fits the observed data (even though that’s the outcome) but instead finding an ap…

Yes, exactly. And it's the compression that makes this happen. If it was just curve-fitting then the LLM really would act like a stochastic parrot, and it wouldn't generalize; also the model would also need to be much much bigger. As it is, the models are very big but they're still tiny compared to the dataset that they compress, so the compression process forces them to approximate a solution to the inverse problem.

And yeah, that means the base model training process is solving the inverse problem of finding an approximation for the processes that generate all human records and also anything else in the dataset with a compressible pattern to discover (weather data, etc). So it's not surprisng that some kind of world model emerges out of this.

Post reply on HN