Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

jasonfantl.com

Post reply on HN
Print view View original

11–20 of 123 posts

Re: What Is Entropy?

#11

Post by glial »

ninetyninenine wrote: post #10
glial wrote: post #8

One thing that helped me was the realization that, at least as used in the context of information theory, entropy is a property of an individual (typically the person receiving a message) and NOT purely of the system or message itself. > entropy quantifies uncertainty This sums it up. Uncertainty is the property of a person and not a system/message. That uncertainty is a function of both a person's model of a system/…

Not true. The uncertainty of the dice rolls is not controlled by you. It is the property of the loaded dice itself. Here's a better way to put it. If I roll the dice infinite times. The uncertainty of the outcome of the dice will become evident in the distribution of the outcomes of the dice. Whether you or another person is certain or uncertain of this does not indicate anything. Now when you realize this you'll sta…

I concede that my framing was explicitly Bayesian, but with that caveat, it absolutely is true: your uncertainty is a function of your knowledge, which is a model of the world, but is not equivalent to the world itself.

Suppose I had a coin that only landed on heads. You don't know this and you flip the coin. According to your argument, for the first flip, your entropy about the outcome of the flip is zero. However, you wouldn't be able to tell me which way the coin would land, making your entropy nonzero. This is a contradiction.

Re: What Is Entropy?

#13

Post by DadBase »

My old prof taught entropy with marbles in a jar and cream in coffee. “Entropy,” he said, “is surprise.” Then he microwaved the coffee until it burst. We understood: the universe favors forgetfulness.

Re: What Is Entropy?

#16

Post by gozzoo »

The visualisation is great, the topic is interesting and very well explained. Can sombody recomend some other blogs with similar type of presentation?

Re: What Is Entropy?

#17

Post by empath75 »

glial wrote: post #8

One thing that helped me was the realization that, at least as used in the context of information theory, entropy is a property of an individual (typically the person receiving a message) and NOT purely of the system or message itself. > entropy quantifies uncertainty This sums it up. Uncertainty is the property of a person and not a system/message. That uncertainty is a function of both a person's model of a system/…

> If we're calculating the entropy of dice rolls (where the outcome is the 'message'), and I know the dice are loaded but you don't, my entropy will be lower than yours.

That's got nothing to do with entropy being subjective. If 2 people are calculating any property and one of them is making a false assumption, they'll end up with a different (false) conclusion.

Re: What Is Entropy?

#19

Post by bloppe »

ninetyninenine wrote: post #10
glial wrote: post #8

One thing that helped me was the realization that, at least as used in the context of information theory, entropy is a property of an individual (typically the person receiving a message) and NOT purely of the system or message itself. > entropy quantifies uncertainty This sums it up. Uncertainty is the property of a person and not a system/message. That uncertainty is a function of both a person's model of a system/…

Not true. The uncertainty of the dice rolls is not controlled by you. It is the property of the loaded dice itself. Here's a better way to put it. If I roll the dice infinite times. The uncertainty of the outcome of the dice will become evident in the distribution of the outcomes of the dice. Whether you or another person is certain or uncertain of this does not indicate anything. Now when you realize this you'll sta…

Probability is subjective though, because macrostates are subjective.

The notion of probability relies on the notion of repeatability: if you repeat a coin flip infinite times, what proportion of outcomes will be heads, etc. But if you actually repeated the toss exactly the same way every time, say with a finely-tuned coin-flipping machine in a perfectly still environment, you would always get the same result.

We say that a regular human flipping a coin is a single macrostate that represents infinite microstates (the distribution of trajectories and spins you could potentially impart on the coin). But who decides that? Some subjective observer. Another finely tuned machine could conceivably detect the exact trajectory and spin of the coin as it leaves your thumb and predict with perfect accuracy what the outcome will be. According to that machine, you're not repeating anything. You're doing a new thing every time.

Re: What Is Entropy?

#20

Post by nihakue »

I'm not in any way qualified to have a take here, but I have one anyway:

My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question.

So for example you could observe that a single coin flip is heads, and entropy will help you quantify how many different ways that could have come to pass. e.g. is it a fair coin, a weighted coin, a coin with two head faces, etc. All these possibilities increase the entropy of the system. An arrangement _not_ counted towards the system's entropy is the arrangement where the coin has no heads face, only ever comes up tails, etc.

Related, my intuition about the observation that entropy tends to increase is that it's purely a result of more likely things happening more often on average.

Would be delighted if anyone wanted to correct either of these intuitions.

Post reply on HN