301–310 of 324 posts
Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing. [0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98
Small off topic thing: I recommend removing the bit in the URL from ?si=... and forward, unless you want Google to track every user who clicks this link to your share.
Earlier quoted context omitted.
A maximally efficient compressor for the existing data distribution is not in general (and often will not be) maximally efficient for future data. The former may only be enabled by convenient local optima of the input distribution that a compressor accounting for the latter could not take advantage of. For instance, consider the distribution of strings drawn from the language '0+'. Now consider the same for the langu…
In a world where we see 0+ strings more often the first compressor would be better
Eg: cigarette smoking was extremely common and a recommended social activity in the past, but now it's not.
If you asked someone would they still smoke if cigarettes were harmful, and they tell you that it's an out of distribution question because they don't know a world were cigarettes are harmful, you'd question their intelligence.
We want predictive models to be able to estimate the consequences of different (counter-factual) actions to then pick the best one.
we are blessed to be in an arbitrary field where we are free to steal ideas from other disciplines and find new innovations in our own.
Earlier quoted context omitted.
I do not understand this intuition that "true consciousness has to be random". The things that make me me are highly deterministic! > LLMs do not 'infer' token streams that haven't been trained in their training process While we're at it, this is simply untrue (in-context learning) unless you generalize "token streams" so radically that it could be readily analogized to humans as well.
> The things that make me me are highly deterministic! Are they though? :-) There are some interesting papers in the tissue regeneration space which are working on building tissue (and organs) from stem cells for medical purposes (transplants, injury treatment, Etc.) and one of the things that comes out from that is that a set of stem cells make unique tissue every time in that it's compatible but the fine structure…
edit: And of course, any pattern that you can recognize about yourself, as "you", has to be deterministic by definition of pattern.
Earlier quoted context omitted.
Oh wow, so it worked pretty well on data it hasn't seen. That expected but cool to reproduce. Have you seen this leaderboard of sorts[1], and this proposal to change hutter prize[2]? I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric because file sizes are very concrete. They are already beating shannon's number…
>I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric This is the premise of https://huggingface.co/spaces/Jellyfish042/UncheatableEval
Earlier quoted context omitted.
You're reading this the wrong way I think, citations aren't given because its obviously a pedagogical article about well established stuff. Much like you wouldn't give citations in a blog post explaining calculus.
One could give citations regarding calculus it's pretty interesting. Since it was done twice by both Newton and Liebniz. There must have been cultural developments in the 1660's that demanded calculus be invented.
Earlier quoted context omitted.
One could give citations regarding calculus it's pretty interesting. Since it was done twice by both Newton and Liebniz. There must have been cultural developments in the 1660's that demanded calculus be invented.
Yeah I agree, I was just saying I really doubt the author of that blog post was trying to take credit for the ideas they present.
Earlier quoted context omitted.
This feels like an AI comment, but I feel compelled to respond. Recursion is not needed when you regress system components to its foundamental representations, which can be derived without any recursion involved. In the thread example, I don't need to recursively determine how to compress how a planet orbits a mass. I only need to 'jump to the end' by define the rules of gravity and the mass/velocity of the bodies. W…
I am flattered that my comment was equated to AI content. English is not even my native language. I will take that as a compliment. Apparently you misunderstood. What made you think I claimed recursion as the key to better intelligence? Clearly you will have noticed that I agree with compression and merely remind one not to forget "recomposition". But once you add recursion to the "mix", we immediately enter the doma…
very odd. hopefully it will go away soon.
Earlier quoted context omitted.
Oh wow, so it worked pretty well on data it hasn't seen. That expected but cool to reproduce. Have you seen this leaderboard of sorts[1], and this proposal to change hutter prize[2]? I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric because file sizes are very concrete. They are already beating shannon's number…
>I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric This is the premise of https://huggingface.co/spaces/Jellyfish042/UncheatableEval
Which implies this would probably also hold for the larger models, which are sadly not included in the leaderboard.
Earlier quoted context omitted.
LLMs are both the best compression and prediction algorithm for English text.
Probably not if you account for algorithm size.
For example, LLM pretraining datasets are on the order of tens or hundreds of terabytes. If an LLM-based code for that data is ~twice as efficient as gzip, you could afford to transmit the weights of even a very large LLM and still come out ahead.
In other words: LLMs actually are excellent compressors of their training sets in the formal information-theoretic sense.