Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compression is prediction

ngrok.com

Post reply on HN
Print view View original

251–260 of 324 posts

Re: Compression is prediction

#251

Post by mainmailman »

detourdog wrote: post #137
bonoboTP wrote: post #107

Earlier quoted context omitted.

I'm of two minds here. The pro is that the "you could have invented this" walkthrough from first principles is more engaging than "and then so and so introduced this term in 1972 and the definition is such and such". This style is a reaction to that boring and dry teaching style and tries to push towards what eg Feynman pointed at in the Brazil critique. The con is that you don't get to understand and see any of the…

Disconnecting idea development from it's historic development is a disservice to the audience that may want to dig deeper.

Have any examples of a blog post that meets your standard?

Re: Compression is prediction

#252

Post by tulio_ribeiro »

zahrevsky wrote: post #7
sheeeeesh wrote: post #3

Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing. [0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98

I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.

In the article, she credits the 2023 DeepMind paper Language Modeling Is Compression as the source. She also links to this post from 2015: https://colah.github.io/posts/2015-09-Visual-Information/

Re: Compression is prediction

#253

Post by adamgordonbell »

speedgoose wrote: post #246
adamgordonbell wrote: post #29

Earlier quoted context omitted.

And also the LLM version, and LLMZip https://bellard.org/ts_zip/ https://arxiv.org/abs/2306.04050

I tried to reproduce those results, at least in terms of compression ratios, not speed. However I would say that testing on alice29, enwiki8, text8 data is kinda cheating. Alice in Wonderland and Wikipedia are very likely part of the training data of the LLM models used there. So I tried on HN comments from a few days ago, extracted from the text column of the public HN bigquery dataset. Using RWKV v7 0.1B instead of…

Oh wow, so it worked pretty well on data it hasn't seen. That expected but cool to reproduce.

Have you seen this leaderboard of sorts[1], and this proposal to change hutter prize[2]?

I think it's a really clever idea that you could measure an LLM's prediction abilities and language understanding by some sort of held-out compression metric because file sizes are very concrete. They are already beating shannon's numbers using a human prediction for compression, from what i can see.

https://github.com/hkust-nlp/llm-compression-intelligence

https://gwern.net/hutter-prize

Re: Compression is prediction

#254

Post by rcxdude »

detourdog wrote: post #239
rcxdude wrote: post #229

Earlier quoted context omitted.

Often the most straightforward way to walk through an idea while teaching it is not the same order that the ideas were developed, and might not even use the same set of ideas in building up to it, so it can be tricky to get both the best explanation of the idea and the historical context in at the same time without making things more confusing.

I like to see ideas presented as the evolved. Each solution is developed as a perceived reaction to the shortcomings of the previous. This becomes a contrast and comparison as to why one idea is appropriate for a particular context.

They're both useful lenses, the evolution just tends to be a bit longer and messier.

Re: Compression is prediction

#255

Post by zeroq »

I've been saying this for years - the best way to wrap your head around AI and LLMs are to think about them as "a whole internet wrapped into a single zip archive, with immensely clever solution to query the data".

That's it.

Once you accept this mental model the implications are staggering.

AI is not "thinking", and it doesn't know the answer to your question because it's smart, but because it has been asked thousand of times over the internet, and it simply provides you with an already existing answer. That code it made for you? It already lived somewhere on GitHub.

But then you have to ask yourself - if we surrender to the AI, who will produce new content 10-15 years from now? If we all pivot from programming to prompt engineering, who will come up with novel solutions?

Re: Compression is prediction

#256

Post by detourdog »

rcxdude wrote: post #254
detourdog wrote: post #239

Earlier quoted context omitted.

I like to see ideas presented as the evolved. Each solution is developed as a perceived reaction to the shortcomings of the previous. This becomes a contrast and comparison as to why one idea is appropriate for a particular context.

They're both useful lenses, the evolution just tends to be a bit longer and messier.

Not by definition. That is one general perception. I'm sure they both have a place and we all have our own preferences.

Re: Compression is prediction

#257

Post by becarlos »

znnajdla wrote: post #245

Intuitively, the idea makes sense to me. You can only compress something when you reduce the content to “what matters” in it. And understanding “what matters” is to understand the patterns in the data. Understanding the patterns in the data IS intelligence. There's an important consequence here which I take as a lesson in life and business: it is worth optimizing a process or a workflow in your life or business even…

  "To optimize, it is the only way to truly understand it."
Mostly.

Re: Compression is prediction

#258

Post by detourdog »

wpietri wrote: post #249
detourdog wrote: post #137

Earlier quoted context omitted.

Disconnecting idea development from it's historic development is a disservice to the audience that may want to dig deeper.

If only there were some sort of way for a reader to dig deeper on a topic without a writer having to spoon-feed them the entire history of everything! It's wild to me what people here expect out of something they got for free and that was offered as a gift.

Mentioning any idea disconnected from its roots may not provide the terms needed to search.

Snarky replies always appreciated

Re: Compression is prediction

#259

Post by qarl2 »

Consider:

If you want to record the motion of the planets, naively you have large tables of positions.

To compress that, you may smoothly interpolate sparse positions.

To compress that, you encode the laws of gravity and simulate from a starting state.

Compression is literally understanding.

Re: Compression is prediction

#260

Post by detourdog »

mainmailman wrote: post #251
detourdog wrote: post #137

Earlier quoted context omitted.

Disconnecting idea development from it's historic development is a disservice to the audience that may want to dig deeper.

Have any examples of a blog post that meets your standard?

They are not blog posts but it is a presentation of technical development that meets my standard.

james burke connections https://www.youtube.com/watch?v=XetplHcM7aQ&pp=ygUXamFtZXMgY...

Tim Hunkin Secret Life of Machines

https://www.youtube.com/watch?v=CJlrbMHLBd4&list=PLtaR0lZhSy...

I have read plenty of blog posts that meet my standard. If I can think of one I will post it.

Post reply on HN