Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

201–210 of 324 posts

Re: Compression is prediction

#201

Post by zahlman »

AnotherGoodName wrote: post #127
throwaway_7274 wrote: post #5

This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can emerge from that process!

Prediction is literally what's allowed computers to make amazingly creative chess and go moves that a human would never have thought of. More generally prediction allows you to path find towards a solution to reach some goal that no human might ever see. If someone states "LLMs can't have new ideas because they only predict" you have to laugh. Prediction literally enables new ideas as you use those predictions to pat…

Being able to read out positions (minimax, alpha-beta pruning etc.) is not "prediction".

Re: Compression is prediction

#202

Post by vrighter »

caust1c wrote: post #51

Compression is not prediction, it is recall. Can we make predictions based on compression? Absolutely. Is memory encoded into physical neurons technically compression? I would argue also yes. However, going from compression to prediction is a large jump that is unsubstantiated by this article and based on the claim that probabilistic recall is also prediction. Two perfect counterpoints to this are markets and weather…

Statistical compressors are all about prediction. They can trivially be reversed to make a generator (not as good as an LLM, but only because it would be a much much much smaller model). And an llm can trivialyl be plopped in there to replace the whole thing.

And a counterpoint to your counterpoint: If it was raining 10 seconds ago, it is very likely that it is still raining right now. There... you can make a prediction about weather based on historical data. It's like perlin noise, random but with discernible patterns

Re: Compression is prediction

#203

Post by zahlman »

versteegen wrote: post #122
nonameiguess wrote: post #45

Earlier quoted context omitted.

There's another element to this that I almost never see discussed. Ideas are not facts. Neither LLMs nor humans can generate new knowledge , as opposed to ideas, by thinking alone. Physical investigation and experimentation is necessary. The exception being pure mathematics since it exists solely in the realm of ideas. I'm willing to call that knowledge, but it's still a distinction, the old analytic/synthetic dichot…

That conception of knowledge is interesting, but I think using the label 'knowledge' for it is very problematic, it's too far from common definitions. The fact that you have to carve out an exception for mathematics already shows there's a problem. Because if maths, shouldn't thought experiments also produce new knowledge? You're excluding special and general relativity. It seems to me that what the concept actually…

> The fact that you have to carve out an exception for mathematics already shows there's a problem.

I don't see how that's an exceptional case. Mathematical theories may not correspond to physical objects, but they still have to be tested for consistency, logical validity etc. As for relativity, one could argue that Einstein's theories were indeed not "knowledge" until their predictions were confirmed (like, say, the gravitational lensing thing). The Michelson–Morley experiment preceded Einstein, but it had to be checked that the theory was consistent with that, and so on and so forth.

Re: Compression is prediction

#204

Post by foo42 »

chermi wrote: post #48
farfatched wrote: post #36

This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University. > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine l…

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

Perhaps tangential to your point, but I often write blog posts (although finish and publish far fewer than I start) where I write about something as it has occurred to me, informed by things I've absorbed no doubt, but without specific research. In such cases I explicitly avoid searching out prior work as a) seeing that something is well discussed and explored can take away the motivation to explore (in the same way reading puzzle solutions before starting might), and b) to avoid having green shoots of ideas shaped by the current of existing consensus. Now that doesn't mean I don't come back after doing my own thinking to see what the more well developed literature of people cleverer than me, who've thought far longer than me think; I just don't want to snuff out my own exploration at the start.

As I say most of these I never publish as I'm mainly using writing as a vehicle for thought, but when I do I'm never sure how to flag them. I don't want (imaginary, lets be honest) readers thinking I'm deluded into thinking I've found something new. I want to come up with a tag I can put on them which adds a pithy disclaimer card at the top or something so I feel more comfortable publishing them.

Re: Compression is prediction

#205

Post by cubefox »

vatsachak wrote: post #173
cubefox wrote: post #165

Earlier quoted context omitted.

The fact that the best compression algorithms are, in the LLM setting, still catastrophically bad at prediction suggests that the overlap between compression and prediction is weak at best.

https://bellard.org/ts_zip/

This shows that prediction algorithms (AI models) are also very good at compression, but compression algorithms (like the ones used in gzip) are not likewise very good at prediction. Which is evidence that compression is necessary but not sufficient for prediction.

Re: Compression is prediction

#207

Post by blobbers »

AlotOfReading wrote: post #124
versteegen wrote: post #117

Earlier quoted context omitted.

There is a distinction between a compressor for a fixed dataset and one for an unknown population from which we have a sample. The optimal compressor for the sample may be the single best guess for the population, but that's not what Solomonoff induction does. It begins with a prior that allows all possible programs, and it never assigns all probability to the single optimal compressor, so it has no problem with the…

I think it's better to think of the hutter prize as a challenge to see how close people can get to efficiently approximating AIXI without the exponential cost of AIXItl. The fact that winners don't generalize well is just that they're not great approximations, because the behavior is different in the limit.

The winner generalize really well. They're just incredibly slow.

Most of the neural compressors are trying to squeeze every last bit out by sacrificing runtime. You can speed them up a fair bit and they're pretty good. fast-cmix with a small priming corpus will probably be about twice as compact as something like lzma or brotli.

Re: Compression is prediction

#208

Post by adrianN »

hnfong wrote: post #179
adrianN wrote: post #146

Earlier quoted context omitted.

I'm not aware of a better definition of "understanding" that would allow me to tell whether some system "understands" some other system. Do you happen to know one?

See: A. M. Turing (1950) Computing Machinery and Intelligence. Mind 49: 433-460. I mean, my interpretation is that the question Turing tried to answer is equivalent to "How can we determine whether machines understand humans/human thought?" This only works when both systems can talk about pretty much arbitrary things, but if you want a more general method for less complex systems, perhaps having one system simulate a…

There are many people who would claim that passing the Turing test is insufficient to show "understanding" (compare for example the Chinese Room thought experiment).

Re: Compression is prediction

#209

Post by WithinReason »

bonoboTP wrote: post #106
farfatched wrote: post #96

Earlier quoted context omitted.

I don't think this is a fair critique. The author of the post uses standard terminology like entropy coding and arithmetic coding, and cited a paper "in 2023, Google DeepMind released a paper arguing that language modeling and compression are two views of the same thing" which discusses it further. This blog post is great. Well explained, and clearly took a lot of effort. I don't interpret it as them claiming to have…

Citing 2023 makes it seem like this is newer than it is. Compression, prediction and intelligence have long been known to be deeply connected.

At least since the 1960s: https://en.wikipedia.org/wiki/Solomonoff%27s_theory_of_induc...

Re: Compression is prediction

#210

Post by larodi »

‘ I was reading about compression recently when I stumbled upon something crazy: that compressors and LLMs are, at their core, trying to solve the exact same problem.’

This reads as written by someone who just happened to understand what LLMs do, so I totally fail to understand how anything further said can have any real credibility…

As a matter of fact the best compression by Fabrice Bellard’s models have been achieved with NNs long before LLMs.

And also MP3 and MPEG in general are very apparent neural networks, yet not deep as in modern VLMs

Post reply on HN