Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

johncarlosbaez.wordpress.com

Post reply on HN
Print view View original

181–190 of 221 posts

Re: What Is Entropy?

#181

Post by petsfed »

kgwgk wrote: post #118
petsfed wrote: post #33

Earlier quoted context omitted.

And this is what I prefer too, although with the clarification that its the number of ways that a system can be arranged without changing its macroscopic properties . Its, unfortunately, not very compatible with Shannon's usage in any but the shallowest sense, which is why it stays firmly in the land of physics.

> not very compatible with Shannon's usage in any but the shallowest sense The connection is not so shallow, there are entire books based on it. “The concept of information, intimately connected with that of probability, gives indeed insight on questions of statistical mechanics such as the meaning of irreversibility. This concept was introduced in statistical physics by Brillouin (1956) and Jaynes (1957) soon after…

I don't dispute that the math is compatible. The problem is the interpretation thereof. When I say "shallowest", I mean the implications of each are very different.

Insofar as I'm aware, there is no information-theoretic equivalent to the 2nd or 3rd laws of thermodynamics, so the intuition a student works up from physics about how and why entropy matters just doesn't transfer. Likewise, even if an information science student is well versed in the concept of configuration entropy, that's 15 minutes of one lecture in statistical thermodynamics. There's still the rest of the course to consider.

Re: What Is Entropy?

#182

Post by _hark »

bubblyworld wrote: post #158
_hark wrote: post #150

Earlier quoted context omitted.

I think what you're getting at is the construction of the sample space - the space of outcomes over which we define the probability measure (e.g. {H,T} for a coin, or {1,2,3,4,5,6} for a die). Let's consider two possibilities: 1. Our sample space is "incomplete" 2. Our sample space is too "coarse" Let's discuss 1 first. Imagine I have a special die that has a hidden binary state which I can control, which forces the…

Right, this is exactly what I'm getting at - learning a distribution over a fixed sample space can be done with Bayesian methods, or entropy-based methods like the OP suggested, but I'm wondering if there are methods that can automatically adjust the sample space as well. For well-defined mathematical problems like dice rolling and fixed classical mechanics scenarios and such, you don't need this I guess, but for any…

We could discuss completeness of the sample space, and we can also discuss completeness of the hypothesis space.

In Solomonoff Induction, which purports to be a theory of universal inductive inference, the "complete hypothesis space" consists of all computable programs (note that all current physical theories are computable, so this hypothesis space is very general). Then induction is performed by keeping all programs consistent with the observations, weighted by 2 terms: the programs prior likelihood, and the probability that program assigns to the observations (the programs can be deterministic and assign probability 1).

The "prior likelihood" in Solomonoff Induction is the program's complexity (well, 2^(-Complexity), where the complexity is the length of the shortest representation of that program.

Altogether, the procedure looks like: maintain a belief which is a mixture of all programs consistent with the observations, weighted by their complexity and the likelihood they assign to the data. Of course, this procedure is still limited by the sample/observation space!

That's our best formal theory of induction in a nutshell.

Re: What Is Entropy?

#183

Post by lachlan_gray »

strogonoff wrote: post #120
bee_rider wrote: post #70

Earlier quoted context omitted.

He was really brilliant, made contributions all over the place in the math/physics/tech field, and had a sort of wild and quirky personality that people love telling stories about. A funny quote about him from a Edward “a guy with multiple equations named after him” Teller: > Edward Teller observed "von Neumann would carry on a conversation with my 3-year-old son, and the two of them would talk as equals, and I somet…

Are there many von-Neumann-like multidisciplinaries nowadays? It feels like unless one is razor sharp fully into one field one is not to be treated seriously by those who made careers in it (and who have the last word on it).

IMO they do exist, but the popular attitude that it's not possible anymore is the issue, not a lack of genius. If everyone has a built in assumption that it can't happen anymore, then we will naturally prune away social pathways that enable it.

Re: What Is Entropy?

#184

Post by cubefox »

bglazer wrote: post #168
cubefox wrote: post #130

Earlier quoted context omitted.

https://emilkirkegaard.dk/en/2022/11/a-theory-of-ashkenazi-g...

Emil Kirkegaard is a self-described white nationalist eugenicist who thinks the age of consent is too high. I wouldn't trust anything he has to say.

> Emil Kirkegaard is a self-described white nationalist

That's simply a lie.

> who thinks the age of consent is too high

Too high in which country? Such laws vary strongly, even by US state, and he is from Denmark. Anyway, this has nothing to do with the topic at hand.

Re: What Is Entropy?

#185

Post by marcosdumay »

glial wrote: post #18

I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…

> he glosses over this

All of information theory is relative to the channel. This bit is well communicated.

What he glosses over is the definition of "channel", since it's obvious for electromagnetic communications.

Re: What Is Entropy?

#186

Post by marcosdumay »

kragen wrote: post #145
glial wrote: post #18

I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…

shannon entropy is subjective for bayesians and objective for frequentists

The entropy is objective if you completely define the communication channel, and subjective if you weave the definition away.

Re: What Is Entropy?

#187

Post by Agentus »

canjobear wrote: post #78
glial wrote: post #18

I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…

What's often lost in the discussions about whether entropy is subjective or objective is that, if you dig a little deeper, information theory gives you powerful tools for relating the objective and the subjective. Consider cross entropy of two distributions H[p, q] = -Σ p_i log q_i. For example maybe p is the real frequency distribution over outcomes from rolling some dice, and q is your belief distribution. You can…

Correct anything thats wrong here. Cross entropy is the comparison of two distributions right? Is the objectivity sussed out in relation to the overlap cross section. And is the subjectivity sussed out not on average but deviations on average? Just trying to understand it in my framework which might be wholly off the mark.

Re: What Is Entropy?

#188

Post by prof-dr-ir »

kgwgk wrote: post #152
prof-dr-ir wrote: post #149

Earlier quoted context omitted.

> is far from “being it” as a definition of the concept of entropy in physics. I simply do not understand why you say this. Entropy in physics is defined using exactly the same equation. The only thing I need to add is the choice of probability distribution (i.e. the choice of ensemble). I really do not see a better "definition of the concept of entropy in physics". (For quantum systems one can nitpick a bit about de…

I’d say that the concept of entropy “in physics” is about (even better: starts with) the choice of a probability distribution. Without that you have just a number associated with each probability distribution - distributions without any physical meaning so those numbers won’t have any physical meaning either. But that’s fine, I accept that you may think that it’s just a little detail. (Quantum mechanics has no myster…

Sarcasm aside, I really do not think you are making much sense.

Obviously one first introduces the relevant probability distributions (at least the micro-canonical ensemble). But once you have those, your comment still does not offer a better way to introduce entropy other than what I wrote. What did you have in mind?

In other words, how did you think I should change this part of my course?

Re: What Is Entropy?

#189

Post by anthk »

cubefox wrote: post #184
bglazer wrote: post #168

Earlier quoted context omitted.

Emil Kirkegaard is a self-described white nationalist eugenicist who thinks the age of consent is too high. I wouldn't trust anything he has to say.

> Emil Kirkegaard is a self-described white nationalist That's simply a lie. > who thinks the age of consent is too high Too high in which country? Such laws vary strongly, even by US state, and he is from Denmark. Anyway, this has nothing to do with the topic at hand.

In Spain used to be as low as 13 a few decades ago; but that law was obviously written before the rural exodus of inner Spain into the cities (from the 60's to almost the 80's), as children since early puberty got to work/help in the farm/fields or at home and by age 14 they had far more duties and accountabilities than today. And yes, that yielded more maturity.

Thus, the law had to be fixed for more urban/civilized times up to 16. Altough depending on the age/mentality closeness (such as 15-19 as it happened with a recent case), the young adult had its charges totally dropped.

Re: What Is Entropy?

#190

Post by kragen »

marcosdumay wrote: post #186
kragen wrote: post #145

Earlier quoted context omitted.

shannon entropy is subjective for bayesians and objective for frequentists

The entropy is objective if you completely define the communication channel, and subjective if you weave the definition away.

the subjectivity doesn't stem from the definition of the channel but from the model of the information source. what's the prior probability that you intended to say 'weave', for example? that depends on which model of your mind we are using. frequentists argue that there is an objectively correct model of your mind we should always use, and bayesians argue that it depends on our prior knowledge about your mind
Post reply on HN