there are many instances of claude in here, so not sure what that disclaimer was about.
Maybe the author has been overly influenced by Claude's writing style?
Connections in Math: the two kinds of random
11–20 of 39 posts
Re: Connections in Math: the two kinds of random
#12This is under-taught in comp sci so I'll say it in all caps for visibility. THERE IS NO KNOWN GENERIC WAY TO MEASURE INFORMATION ENTROPY!!!!
Seriously. There isn't. You might say 'but measure the number of 1's vs 0's' and i'll just reply with '101010101010 repeated'. So you up it and start saying ok maybe measure frequencies of pairwise bits then and i just reply by cycling 00,01,10,11 endlessly etc. Frequency counts and any other measure you can think of doesn't actually work.
The root definition will state it's just based on the properties of the prediction but... what model do you use for the prediction? This gets to the heart of the matter;
The measure of information entropy itself is actually equivalent to creating on omniscient oracle. It requires being able to ask the question "Oh great oracle, what's the shortest program that can reproduce X"? or equivalently "What's the probability of this data assuming i always have the most perfect model to predict it?". You then calculate entropy based on that oracles answer.
Kolmorogorov, AI and information entropy are all pointing to the same thing. The reason people get confused by the fact that the frequencies of base 10 digits is the same for digits of pi and a truly random sequence is just because the counts of base 10 digits isn't a measure of entropy at all. Measuring entropy literally requires an omniscient oracle.
Re: Connections in Math: the two kinds of random
#13The thing that frustrates me about this argument is that there is no shortest program that produces pi. You need a computer to run it, which is massive non compressed data, or a human to calculate stuff, an uncountable amount of entropy. I see that the irrational pi has a smooth distribution of digits and a file full of zeroes is compressible, but they are both sort of magically part of a world that does not run prog…
I think that does not hold, Kolmogorov complexity is measured relative to a pre-defined universal machine for everything. The machine is not counted in the description of π, for the same reason a book's length isn't measured by including the size of the reader. You fix one interpreter, then ask "how long is the shortest input that makes something?" The interpreter is a constant — the same constant for π, for the rand…
Sure, but spigot is a library, not the interpreter. The author’s program is short because it hides that.
Re: Connections in Math: the two kinds of random
#14Lately I’ve felt Kolmogorov complexity is an unfair measurement because it takes for granted your underlying programming language as treats it as zero cost. In theory you could create a custom language and embed the program as data and “compress” a large random sequence with a better Kolmogorov complexity for that specific language than Pi, simply by not exposing the ability in the language to even work with Pi. I th…
You always include the measurement of things needed to run the program too.
It's a bit like how benchmarks of compression utilities should always include the size of the utility itself. Otherwise someone can just submit a program with a dictionary of 256 common benchmark files for compression and claim "it compresses them to a single byte" :)
Re: Connections in Math: the two kinds of random
#15there are many instances of claude in here, so not sure what that disclaimer was about.
True. There is no shame in AI writing or human writing. Content is what matters. It appears the author is trying to fake genuinity in a fundamentally wrong way and yet doesn't even deliver on it.
Re: Connections in Math: the two kinds of random
#16there are many instances of claude in here, so not sure what that disclaimer was about.
how do you know?
Re: Connections in Math: the two kinds of random
#17Lately I’ve felt Kolmogorov complexity is an unfair measurement because it takes for granted your underlying programming language as treats it as zero cost. In theory you could create a custom language and embed the program as data and “compress” a large random sequence with a better Kolmogorov complexity for that specific language than Pi, simply by not exposing the ability in the language to even work with Pi. I th…
His definition of Kolmogorov complexity is a bit loose. The rigorous definition uses Turing machines (or Minsky, or Post, or some sort of lambda expression, etc.) so the size is something specific. Different versions of complexity defined this way may give different values but have the same properties and asymptotics so one might just as well stick with the Turing kind. Chaitin's theorem (about the limit of Kolmogoro…
it's not just that they have the same asymptomatic. once you do the radix conversion (e.g. base 10 has log(10)/log(2) times more symbols), any 2 definitions are only off from each other by a constant
Re: Connections in Math: the two kinds of random
#18there are many instances of claude in here, so not sure what that disclaimer was about.
Yep. I'm already not a huge fan of reading AI writing, and adding an obviously false disclaimer at the top is ridiculous.
Re: Connections in Math: the two kinds of random
#19This is under-taught in comp sci so I'll say it in all caps for visibility. THERE IS NO KNOWN GENERIC WAY TO MEASURE INFORMATION ENTROPY!!!! Seriously. There isn't. You might say 'but measure the number of 1's vs 0's' and i'll just reply with '101010101010 repeated'. So you up it and start saying ok maybe measure frequencies of pairwise bits then and i just reply by cycling 00,01,10,11 endlessly etc. Frequency counts…
> This is under-taught in comp sci so I'll say it in all caps for visibility. THERE IS NO KNOWN GENERIC WAY TO MEASURE INFORMATION ENTROPY!!!!
Both your comment and the OP seem to conflate a distribution with a sample from a distribution.
A distribution over strings has entropy. A single string can have a Kolmogorov complex but does not have entropy in the Shannon sense per se.
When you write a compression algorithm, you are often effectively building an algorithm to look at a single sample from an unknown distribution (the input data) and trying to invent a distribution that is both easy to describe and that produces the input data with respectably large probability. And then you output the description of the distribution and enough data to identify the sample in question, and the latter takes space that is roughly the entropy of the distribution you just made up.
> "What's the probability of this data assuming i always have the most perfect model to predict it?". You then calculate entropy based on that oracles answer.
Careful here. The most perfect model may be the one that outputs the data in question with probability one, and the entropy is exactly zero.
If you want information theory to give you meaningful answers, you need to ask it more carefully specified questions.
Re: Connections in Math: the two kinds of random
#20This is under-taught in comp sci so I'll say it in all caps for visibility. THERE IS NO KNOWN GENERIC WAY TO MEASURE INFORMATION ENTROPY!!!! Seriously. There isn't. You might say 'but measure the number of 1's vs 0's' and i'll just reply with '101010101010 repeated'. So you up it and start saying ok maybe measure frequencies of pairwise bits then and i just reply by cycling 00,01,10,11 endlessly etc. Frequency counts…
Measuring information entropy seems to have parallels with the incompleteness theorem and with the hard problem. As we approach the seams (so to speak) in our ability to model the system in its own terms, our otherwise precise maps become distorted, things get self-referential and paradoxical.