Bayes, Bits and Brains

#1

Post by cgadski » Thu, Aug 28, 2025, 1:51 PM UTC

Bayes, Bits and Brains
bayesbitsbrains.github.io

Re: Bayes, Bits and Brains

#2

Post by AdieuToLogic » Mon, Sep 01, 2025, 2:04 AM UTC

Sometimes it is best to remember that you are a person and are not in competition with algorithms.

The author labels the first riddle thusly:

  Intelligence test

And reinforces this with the first sentence:

  Test your intelligence with the following widget!

If trying to solve it is fun and the faux implication immaterial, awesome. However, if the expressed characterization of:

  Don't feel bad if a machine beats you ...

Is bothersome, remember that all of this is just one person's way of shaping your experience such that continued engagement is likely.

Re: Bayes, Bits and Brains

#3

Post by JoshCole » Mon, Sep 01, 2025, 4:19 AM UTC

> We are now getting an equivalent definition of what neural nets are being trained for! LLMs are trained to compress the internet as much as possible!

Nice payoff. Others have also called out the relationship to compression (https://www.youtube.com/watch?v=AKMuA_TVz3A).

Re: Bayes, Bits and Brains

#4

Post by _0ffh » Mon, Sep 01, 2025, 8:48 AM UTC

JoshCole wrote: post #3

> We are now getting an equivalent definition of what neural nets are being trained for! LLMs are trained to compress the internet as much as possible! Nice payoff. Others have also called out the relationship to compression ( https://www.youtube.com/watch?v=AKMuA_TVz3A ).

The relationship has been thought about for a long time. In 2006 it even led to the creation of the Hutter Prize, with around 38k€ payed out so far.

Re: Bayes, Bits and Brains

#5

Post by CuriouslyC » Mon, Sep 01, 2025, 11:19 AM UTC

JoshCole wrote: post #3

> We are now getting an equivalent definition of what neural nets are being trained for! LLMs are trained to compress the internet as much as possible! Nice payoff. Others have also called out the relationship to compression ( https://www.youtube.com/watch?v=AKMuA_TVz3A ).

Framing it as compression is reductive (intended). Yes compression of information is a proxy measure of Kolmogorov complexity, however it's really more accurate to say you're accurately mapping the conditional probability distribution, since it's a stochastic machine that produces samples from a distribution, not a literal compressed representation of anything (you have to do work to extract this stuff and it's not 100% in all cases).

Re: Bayes, Bits and Brains

#6

Post by vasekrozhon » Thu, Sep 04, 2025, 9:39 AM UTC

AdieuToLogic wrote: post #2

Sometimes it is best to remember that you are a person and are not in competition with algorithms. The author labels the first riddle thusly: Intelligence test And reinforces this with the first sentence: Test your intelligence with the following widget! If trying to solve it is fun and the faux implication immaterial, awesome. However, if the expressed characterization of: Don't feel bad if a machine beats you ... I…

Hi, the author here. It was meant as a light joke, and I totally agree that we should not feel bad if an AI can predict the next letter better than us. Maybe I should rephrase it to something like "IQ test for AIs" to hint a bit more about what's to come -- this is what we train the AIs for to be intelligent.

Re: Bayes, Bits and Brains

#7

Post by vasekrozhon » Thu, Sep 04, 2025, 9:47 AM UTC

JoshCole wrote: post #3

> We are now getting an equivalent definition of what neural nets are being trained for! LLMs are trained to compress the internet as much as possible! Nice payoff. Others have also called out the relationship to compression ( https://www.youtube.com/watch?v=AKMuA_TVz3A ).

Hi, the author here.

@JoshCole Thanks -- I also like that lecture by Ilya, added it to the resources now.

@0ffh -- Yeah I hope the page does not come across as me trying to claim some new revolutionary insights. People like Ilya have been talking about this years ago - I am just trying to package it into a hopefully more accessible format.