Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

271–280 of 324 posts

Re: Compression is prediction

#271

Post by AnotherGoodName »

zahlman wrote: post #201
AnotherGoodName wrote: post #127

Earlier quoted context omitted.

Prediction is literally what's allowed computers to make amazingly creative chess and go moves that a human would never have thought of. More generally prediction allows you to path find towards a solution to reach some goal that no human might ever see. If someone states "LLMs can't have new ideas because they only predict" you have to laugh. Prediction literally enables new ideas as you use those predictions to pat…

Being able to read out positions (minimax, alpha-beta pruning etc.) is not "prediction".

Tree pruning needs prediction to work and is merely part of the search of predicted spaces.

The prediction is in the state evaluator that predicts how much closer to victory any position is.

Chess and go absolutely have predictions being made by their value nets.

Re: Compression is prediction

#272

Post by elendilm »

qarl2 wrote: post #259

Consider: If you want to record the motion of the planets, naively you have large tables of positions. To compress that, you may smoothly interpolate sparse positions. To compress that, you encode the laws of gravity and simulate from a starting state. Compression is literally understanding.

Nice one.

Yes. Compression constitutes necessary conditions for understanding. But don't forget recomposition as well.

Add recursion to the mix and it results in recursive compression and recomposition - making understanding itself a self existing entity.

This is as close to an understanding god you can rigorously get to.

Re: Compression is prediction

#273

Post by bonoboTP »

wpietri wrote: post #267
detourdog wrote: post #258

Earlier quoted context omitted.

Mentioning any idea disconnected from its roots may not provide the terms needed to search. Snarky replies always appreciated

Your theory is that anybody who writes anything is obligated to make sure you can find any related information with one Google search? Again, to me that looks like wanting to be spoon fed. I have no idea why you think the world owes you endless 101-level discourse, but I hope you recognize you're setting yourself up for equally endless disappointment. If you take a little responsibility for your own education, you'll…

The overall message and impression of a text has to give the right picture regarding what's new, where the info comes from, etc. It's not about any kind of Google search. It's not about extra info. It's about intellectual honesty.

Re: Compression is prediction

#274

Post by detourdog »

wpietri wrote: post #267
detourdog wrote: post #258

Earlier quoted context omitted.

Mentioning any idea disconnected from its roots may not provide the terms needed to search. Snarky replies always appreciated

Your theory is that anybody who writes anything is obligated to make sure you can find any related information with one Google search? Again, to me that looks like wanting to be spoon fed. I have no idea why you think the world owes you endless 101-level discourse, but I hope you recognize you're setting yourself up for equally endless disappointment. If you take a little responsibility for your own education, you'll…

My theory is that anyone wishing to elucidate a complex technical concept is better served with contextual details.

I think one method is superior to another.

Anybody, anything, obligated, and any related information are hyperbolic terms I would have avoided. I understand with a weak position people often resort to hyperbole.

I believe adding more terms to be googled is superior to fewer search terms. When working with ideas from first principles sharing the roots and development is superior to making no reference.

I have no idea why you think the world owes me anything. I was describing what I prefer.

I'm not sure why you think I'm not already endlessly disappointed by the state of the world and it's discourse. not having expectations always helps.

I think that if you stop projecting ideals on strangers might help your mood.

Re: Compression is prediction

#275

Post by vatsachak »

cubefox wrote: post #205
vatsachak wrote: post #173

Earlier quoted context omitted.

https://bellard.org/ts_zip/

This shows that prediction algorithms (AI models) are also very good at compression, but compression algorithms (like the ones used in gzip) are not likewise very good at prediction. Which is evidence that compression is necessary but not sufficient for prediction.

LLMs are both the best compression and prediction algorithm for English text.

Re: Compression is prediction

#276

Post by dev_cprice »

zahlman wrote: post #196

The page source appears to contain all the actual text within tags, but structured in a completely illogical way. With JavaScript disabled, there are a bunch of shaded bars where the text should appear, which look like placeholders for something that hasn't loaded yet even though it was there from the beginning. The tags don't even seem to show up in the DOM. (I didn't check closely, but maybe they're embedded in an…

Hey, appreciate the feedback! This is definitely a bug. Will get it fixed asap!

Re: Compression is prediction

#277

Post by chermi »

beagle3 wrote: post #159
bonoboTP wrote: post #106

Earlier quoted context omitted.

Citing 2023 makes it seem like this is newer than it is. Compression, prediction and intelligence have long been known to be deeply connected.

You expect every blog post to find the earliest relevant paper to cite, just so one could look at the year (without reading said paper - which would have made clear that the connection isn’t recent) to assess novelty? I don’t think that’s reasonable. It’s a blog post. If it was, say, a peer reviewed paper by Hinton or LeCunn that fails to cite Schmidhuber, that would be reasonable criticism in my opinion. (Spoiler: t…

I guess it depends on the actual goal of the post. Education? Point people to background. Surely if you have a deep understanding of the subject, you have favored references.

If it's not education, where people are supposed to know everything, who is the audience? If the audience is supposed to know all of the background, then the post is not saying something original to the audience.

I agree the quality of the text itself is good. But I knew the background and I left (well, actually began) already agreeing.

I am happy and want people to write and share stuff. I think it's a really good exercise if only for the author. To be clear, I think the post itself is good except for the gap I mentioned in my original post, which in the grand scheme of things may seem like a nitpick. But if the same type of content makes it to the front page often without me understanding who the audience actually is, I am also allowed to wonder why out loud.

Re: Compression is prediction

#278

Post by chermi »

zahlman wrote: post #191
genxy wrote: post #183

Earlier quoted context omitted.

> You expect every blog post to find the earliest relevant paper to cite This should be expected out of everyone. If you don't respect the reader enough to do this, why should we read your posts? I think papers should be retracted for not citing prior art, even if you weren't aware of it.

If the blog were about calculus, and stated that an elegant proof of the Fundamental Theorem of Calculus could be found in such and such undergraduate textbook, would you be upset that the citation wasn't to either Newton's or Leibniz' work?

I would want them to point to point to what they think is the best resource. (Assuming the point is educating or explaining something to the reader.) Which, If everyone followed, would automatically point back to newton/leibnez.

Re: Compression is prediction

#279

Post by lopsotronic »

WithinReason wrote: post #216
augment_me wrote: post #150

Earlier quoted context omitted.

As much as I want to, I sadly don't think Information Theory makes sense in this setting, and I really wanted to believe this. When Shannon made his theory of information, he was always dealing with informational representations on the abstraction level of bits. At Bell Labs, a lot of the work was on the compression of data for transfer over telephone wires. Entropy coding, later codexes like algorithmic coding, and…

Check out https://en.wikipedia.org/wiki/Kolmogorov_complexity

AKA why Component Content Systems usually fall flat on their face over time.

Re: Compression is prediction

#280

Post by chermi »

rcxdude wrote: post #229
Dylan16807 wrote: post #152

Earlier quoted context omitted.

There's no reason a compelling walkthrough can't throw in historical citations, is there?

Often the most straightforward way to walk through an idea while teaching it is not the same order that the ideas were developed, and might not even use the same set of ideas in building up to it, so it can be tricky to get both the best explanation of the idea and the historical context in at the same time without making things more confusing.

I agree. Simple albeit imperfect solution would be pointing even to a wiki page. That alone would signal acknowledgement/awareness that it's building on other stuff.
Post reply on HN