Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

191–200 of 324 posts

Re: Compression is prediction

#191

Post by zahlman »

genxy wrote: post #183
beagle3 wrote: post #159

Earlier quoted context omitted.

You expect every blog post to find the earliest relevant paper to cite, just so one could look at the year (without reading said paper - which would have made clear that the connection isn’t recent) to assess novelty? I don’t think that’s reasonable. It’s a blog post. If it was, say, a peer reviewed paper by Hinton or LeCunn that fails to cite Schmidhuber, that would be reasonable criticism in my opinion. (Spoiler: t…

> You expect every blog post to find the earliest relevant paper to cite This should be expected out of everyone. If you don't respect the reader enough to do this, why should we read your posts? I think papers should be retracted for not citing prior art, even if you weren't aware of it.

If the blog were about calculus, and stated that an elegant proof of the Fundamental Theorem of Calculus could be found in such and such undergraduate textbook, would you be upset that the citation wasn't to either Newton's or Leibniz' work?

Re: Compression is prediction

#192

Post by vrighter »

woliveirajr wrote: post #33

There is Compression done by Prediction by partial matching [0] There is the Kolmogorov Complexity [1], Normalized Information Distance [2] and Normalized compression distance [3] that correlates those. Finally, there's the Pre-Big Bang Informational Compression and the Delayed Release of Antimatter [4] All big {rabbit/black} holes to lose some time, if you have any. [0] https://en.wikipedia.org/wiki/Prediction_by_pa…

This is exactly why I think they are one and the same. It's relatively trivial to just plop a (lossy) machine learned markov chain instead of one learned (perfectly) from the data into PPM. With zero changes to the rest of the algorithm.

Re: Compression is prediction

#193

Post by userbinator »

chermi wrote: post #48
farfatched wrote: post #36

This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University. > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine l…

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

they discovered from scratch themselves

If you followed the data compression scene in the 80s and early 90s, there were plenty of reinventions of LZ-ish and Huffman-ish algorithms (I also coded my own variant...), and people even tried to patent some of them, so at least for the basics I think it is something that many can discover independently; of course in these times, it's more likely they didn't.

Re: Compression is prediction

#194

Post by ascorbic »

jdthedisciple wrote: post #166

There is a correct sense, but we're sort of garbling concepts here: Predictability is the inverse of information density. Low information density enables high compression, and vice versa. It's called entropy. This is basic information theory to be quite frank..

She covers all of this in the post

Re: Compression is prediction

#195

Post by qsera »

sdenton4 wrote: post #163
ChuckMcM wrote: post #160

Earlier quoted context omitted.

It also helps explain to people that LLMs are as likely as bzip to develop "consciousness".

Let alone a sack of wet, self replicating protein! Just endless copying... How could it ever do anything more?

This self replicating proteins can do a lot of stuff that a program running inside a computer cannot do. Like producing secretions, generate electricity, generate light, generate force etc etc....

Re: Compression is prediction

#196

Post by zahlman »

The page source appears to contain all the actual text within

tags, but structured in a completely illogical way. With JavaScript disabled, there are a bunch of shaded bars where the text should appear, which look like placeholders for something that hasn't loaded yet even though it was there from the beginning. The

tags don't even seem to show up in the DOM. (I didn't check closely, but maybe they're embedded in an inline script.)

This is actively user-hostile. The site is going out of its way to interfere with the most basic possible function of HTML, i.e., the presentation of minimally marked-up plain text. The needless complexity is especially ironic in the context of an article about compression.

Re: Compression is prediction

#197

Post by vrighter »

AnotherGoodName wrote: post #121
ssivark wrote: post #14

Nope; there is a bit more nuance and the distinction is important. Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems . The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same support ! Eg: you observe a rare edge case in your training data and (lossy)…

The record setting file compressors are all very generalizable. This comes about because you score compression by adding the program size (including any training data or dictionary) to the compressed data. If they didn't score it this way you could just ship a dictionary of the test data and your program just prints that. So of course they combine it and effectively measure the Kolmorogov complexity. This means that…

Statistical compression works by learning statistics about the data as compression progresses. This is a(n admittedly very good) implementation of that, but the idea of learning about the data is as old as time (or computers, whichever came last). Pretty much all compressors do this, wither by building a huffmann tree on blocks and shipping the tree allowing the data to be decoded, or by incrementally updating statistics after each token, and using the updated statistics for the next one.

Re: Compression is prediction

#198

Post by vrighter »

Legend2440 wrote: post #17
pjankiewicz wrote: post #10

I was thinking about the same topic and the conclusion can be wrong. LLMs are compressors, but compressors are not LLMs. Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot. Specifically I was thinking about a way to inject knowledge into LLMs training by using statistical properties of text in such a way that you don't have to train the LLM to achieve some lev…

>Mixing this can let you believe that you can use a compressor to do the same thing as LLMs, which you cannot. You can, actually! Any compressor can be losslessly converted into a generator, and vice versa. Traditional compressors like gzip are of course very simple and can only replicate rough patterns from the input. But they are technically doing the same thing.

Actually, it's trivial. I did it for fun once when I was learning about the PPM algorithm. It took about 15 minutes to reverse the whole thing.

Re: Compression is prediction

#199

Post by ascorbic »

chermi wrote: post #48
farfatched wrote: post #36

This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University. > Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and neuroscientists, all studying common problems. Information theory and machine l…

I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply, they want shiny stuff that makes them seem deep. This post references spec…

The first sentence says she came across it when reading about compression. I didn't read that as her claiming to have discovered the idea or that it was a new idea, I read it as "today I learned". I think somebody who was unfamiliar with how compression algorithms or language models work would find this an approachable and interesting introduction. Not everybody studied information theory.

Re: Compression is prediction

#200

Post by ErenayDev »

zahlman wrote: post #196

The page source appears to contain all the actual text within tags, but structured in a completely illogical way. With JavaScript disabled, there are a bunch of shaded bars where the text should appear, which look like placeholders for something that hasn't loaded yet even though it was there from the beginning. The tags don't even seem to show up in the DOM. (I didn't check closely, but maybe they're embedded in an…

I think every website should be support noscript with minimum requirements.
Post reply on HN