Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Entropy explained, with sheep (2016)

aatishb.com

Post reply on HN
Print view View original

101–107 of 107 posts

Re: Entropy explained, with sheep (2016)

#101

Post by ba0bab »

great explanation and visuals. but I do not quite get the way the arrangements of sheep are treated. They seem to be counted in the standard "Unordered Sampling with Replacement" fashion.

>Just as the sheep wander about the plots of land in the farm, these packets of energy randomly shuffle among the atoms in the solid.

this would mean that any "packet" of energy is equally likely to be in any of the buckets and the "packets" are independent of each other. so we have an equal distribution on the product space with 6^6 elements.

but then later

> Now, let’s assume the sheep are equally likely to be in any of these 462 arrangements. (Since they move randomly, there's no reason to prefer one arrangement over another.)

under the prior assumptions these arrangements would not be equally likely. e.g. "all sheeps in plot 1" would be far less likely than "each sheep in a different plot" am I missing something here?

in any case the same conclusions can be drawn in both cases, only that the concentration around 3 is already more pronounced in the "6 sheep, 6 plots" case using the product space model.

Re: Entropy explained, with sheep (2016)

#102

Post by roimor »

kgwgk wrote: post #100
roimor wrote: post #97

I don't really understand this "Entropy Is All About Arrangements" takeaway. A fair coin has higher entropy than a biased one. What are the "arrangements" in this case?

The coins don't have entropy, the sequences they produce do (in information theory sense, this is not about physics). For a long sequence (say 1000), the former will produce close to 500 heads and 500 tails. The latter, assuming one heads are three times as probable as tails, will produce around 750 heads and 250 tails. There are many more different sequences of the first kind.

I'm referring to the entropy of the Bernoulli distribution. If the coin is fair, the entropy is 1 bit... if the coin isn't fair, then the entropy of the distribution is less than 1 bit. I'm having trouble reconciling the information theory way of thinking about entropy as a function of a distribution, with how physicists tend to think of entropy of arrangements.

Re: Entropy explained, with sheep (2016)

#103

Post by PaulDavisThe1st »

jbay808 wrote: post #99
PaulDavisThe1st wrote: post #73

Earlier quoted context omitted.

>why can't you remember the future the way you can remember the past this is getting a bit meta, maybe even off-topic. But I think this is fairly simple to explain: you can't remember states you haven't been in. Generalized memory implies some record of a state that has occured - states that have not occured cannot be remembered. The problem is that memory of types that we are familiar with (human, written, computer)…

Sorry for the late reply. Don't know if you'll still be reading this, but... > This is getting a bit meta, maybe even off-topic. But I think this is fairly simple to explain: you can't remember states you haven't been in. Generalized memory implies some record of a state that has occured - states that have not occured cannot be remembered. In a discussion about the arrow of time, this is somewhat begging the question…

>In a discussion about the arrow of time, this is somewhat begging the question!

Not really. "Been in" isn't meant to imply a temporal relationship. If a system has been in microstates A, B and C, then it can potentially remember states A, B and C. It cannot remember state D. That doesn't stay anything about whether or not A preceeded B, or C preceeded A.

I agree with your last two paragraphs, but not the formulation in the middle para. "you cannot remember the future" ... this just seems like an non-useful observation to me. I think the problem comes from this line:

>We (and other physical systems) are able to form memories in one direction, but not the other.

I think this is wrong. The issue is that our memories are of macrostates, and macrostates are subject to the entropy gradient. If we could form memories of microstates, we'd effectively be able to remember events that had no arrow of time associated with them. But then you say this yourself in your final line. And Maccone's concept seems fine to me (as if that matters :)

Re: Entropy explained, with sheep (2016)

#104

Post by emdubbelyou »

ba0bab wrote: post #101

great explanation and visuals. but I do not quite get the way the arrangements of sheep are treated. They seem to be counted in the standard "Unordered Sampling with Replacement" fashion. >Just as the sheep wander about the plots of land in the farm, these packets of energy randomly shuffle among the atoms in the solid. this would mean that any "packet" of energy is equally likely to be in any of the buckets and the…

I think the confusion is in the way that sheep as a word can be both plural and singular. Specifically, one sheep is as likely to be in any single spot compared to any other single spot. It’s when you get to more than one sheep that you see the distributions

Re: Entropy explained, with sheep (2016)

#105

Post by [deleted] »

roimor wrote: post #102
kgwgk wrote: post #100

Earlier quoted context omitted.

The coins don't have entropy, the sequences they produce do (in information theory sense, this is not about physics). For a long sequence (say 1000), the former will produce close to 500 heads and 500 tails. The latter, assuming one heads are three times as probable as tails, will produce around 750 heads and 250 tails. There are many more different sequences of the first kind.

I'm referring to the entropy of the Bernoulli distribution. If the coin is fair, the entropy is 1 bit... if the coin isn't fair, then the entropy of the distribution is less than 1 bit. I'm having trouble reconciling the information theory way of thinking about entropy as a function of a distribution, with how physicists tend to think of entropy of arrangements.

[deleted]

Re: Entropy explained, with sheep (2016)

#106

Post by kgwgk »

roimor wrote: post #102
kgwgk wrote: post #100

Earlier quoted context omitted.

The coins don't have entropy, the sequences they produce do (in information theory sense, this is not about physics). For a long sequence (say 1000), the former will produce close to 500 heads and 500 tails. The latter, assuming one heads are three times as probable as tails, will produce around 750 heads and 250 tails. There are many more different sequences of the first kind.

I'm referring to the entropy of the Bernoulli distribution. If the coin is fair, the entropy is 1 bit... if the coin isn't fair, then the entropy of the distribution is less than 1 bit. I'm having trouble reconciling the information theory way of thinking about entropy as a function of a distribution, with how physicists tend to think of entropy of arrangements.

There is a relationship between the distribution of x_i and the sequences generated from that distribution x_1, x_2, ..., x_n. If the coin isn't fair there are less arrangements possible. If the coin has two heads there is only one possible sequence.

Re: Entropy explained, with sheep (2016)

#107

Post by matt-attack »

anonytrary wrote: post #93
IIAOPSW wrote: post #76

Earlier quoted context omitted.

I asked the same question to my professor when I took thermodynamics as an undergrad. In that class we are told that a particle in a box of size 2 can be in twice as many places as a particle in a box of size 1. But in real analysis we learn there are just as many numbers between 0 and 1 as there are between 0 and 2. The answer I was given, in true physicists form, is hand-wave it. There's an intuitive notion that tw…

> energy and position come in discrete units This is not true; position/time are not quantized in the standard model. String theory is not canonical. I think a better way to think about it is not in terms of size, but in terms of time. A particle in a bigger box, on average, can go on a random walk for longer without hitting a wall. It will take longer for a particle to sufficiently (arbitrarily close) exhaust the ph…

I thought the Planck constant somehow tied in with the smallest unit of length allowed. Sort of like the pixels of 3-space.
Post reply on HN