Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It took me 10 years to understand entropy

cantorsparadise.com

Post reply on HN
Print view View original

101–110 of 295 posts

Re: It took me 10 years to understand entropy

#101

Post by [deleted] »

ogogmad wrote: post #67
omegalulw wrote: post #65

Earlier quoted context omitted.

> I thought entropy (in the Shannon sense) was a property of discrete and finite probability distributions. It's essentially a measure of how random a sample from such a probability distribution is. Notably, continuous probability distributions don't have meaningful entropy (or in some sense, their entropy is always infinite). True, but for continuous distributions you can use the KL divergence against a uniform dist…

One of the properties of entropy H(X) of a random variable X is that if f is a bijective function then H(f(X)) = H(X). For relative entropy (or "KL divergence" as some people call it), we have that H(X||Y) = H(f(X)||f(Y)). But if you fix Y to have a continuous uniform distribution, then you lose this critical property because f(Y) may no longer have a continuous uniform distribution.

[deleted]

Re: It took me 10 years to understand entropy

#102

Post by kgwgk »

ogogmad wrote: post #67
omegalulw wrote: post #65

Earlier quoted context omitted.

> I thought entropy (in the Shannon sense) was a property of discrete and finite probability distributions. It's essentially a measure of how random a sample from such a probability distribution is. Notably, continuous probability distributions don't have meaningful entropy (or in some sense, their entropy is always infinite). True, but for continuous distributions you can use the KL divergence against a uniform dist…

One of the properties of entropy H(X) of a random variable X is that if f is a bijective function then H(f(X)) = H(X). For relative entropy (or "KL divergence" as some people call it), we have that H(X||Y) = H(f(X)||f(Y)). But if you fix Y to have a continuous uniform distribution, then you lose this critical property because f(Y) may no longer have a continuous uniform distribution.

Apparently this "critical property" is not so important to all the people who use relative entropy as a generalization to a continuous distribution defined on a space with an underlying measure.

Why would they care about arbitrary transformations mapping points in the space to other points in the space?

Re: It took me 10 years to understand entropy

#103

Post by oldsecondhand »

andi999 wrote: post #81
jbay808 wrote: post #73

Earlier quoted context omitted.

> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a ther…

Entropy (differences) are an objective quantity which can be measured, there is no subjectivity about it. It is not which parameters you know it is about which parameters you hold fixed.

I think he is confusing the usage of entropy in physics and computer science. In computer science entropy is conditional probability and depends on what we know about a system.

Re: It took me 10 years to understand entropy

#105

Post by kgwgk »

sytelus wrote: post #79

Anyone who think they understand entropy is living in a state of sin.

For those missing the reference, the original quote is also great:

"Anyone who attempts to generate random numbers by deterministic means is, of course, living in a state of sin." John von Neumann

Re: It took me 10 years to understand entropy

#106

Post by Koshkin »

empiricus wrote: post #90

My understanding of entropy: it is a measure of how big a system (matter + energy from a space region) is, and how much its components have interacted with each other: Entropy ~ log(number of possible system states). As the universe unfolds, systems originally isolated are starting to interact and to form bigger systems, hence the number of possible states increases, and entropy increases too.

Entropy implies that these states are indistinguishable from each other.

Re: It took me 10 years to understand entropy

#107

Post by kgwgk »

oldsecondhand wrote: post #103
andi999 wrote: post #81

Earlier quoted context omitted.

Entropy (differences) are an objective quantity which can be measured, there is no subjectivity about it. It is not which parameters you know it is about which parameters you hold fixed.

I think he is confusing the usage of entropy in physics and computer science. In computer science entropy is conditional probability and depends on what we know about a system.

As it does in physics!

"which parameters [thermodynamic variables] you know" ~ "which parameters [thermodynamic variables] you hold fixed"

(or know in average, like the energy for a system in a heat bath where the temperature is fixed)

https://bayes.wustl.edu/etj/articles/theory.1.pdf

http://nicf.net/articles/thermodynamics-statistical-mechanic...

Re: It took me 10 years to understand entropy

#108

Post by [deleted] »

l33t2328 wrote: post #48
chemmail wrote: post #42

I had an art teacher who was very philosophical. One day he described to the class what entropy was. I took a lot of physics and even astrophysics. Little did i know he had a better conceptual understanding and explanation than i've ever heard before. Too bad i don't remember exactly what he said.

Not to poke holes in your nostalgia, but how do you know he had a great explanation if you don’t remember it after further study?

[deleted]

Re: It took me 10 years to understand entropy

#109

Post by Koshkin »

chemmail wrote: post #42

I had an art teacher who was very philosophical. One day he described to the class what entropy was. I took a lot of physics and even astrophysics. Little did i know he had a better conceptual understanding and explanation than i've ever heard before. Too bad i don't remember exactly what he said.

In art, a high-entropy painting is one that would be hard to tell apart from similar paintings, one example being paintings created by simply splattering paints all over the canvas.

Re: It took me 10 years to understand entropy

#110

Post by Gravityloss »

Gas molecules in a box - entropy seems quite straighforward there. An even distribution is the most likely state and has the highest entropy.

In space, at large scales, gravity starts dominating - so stars and planets are actually a higher likelihood state than an even distribtion.

Isn't this just about statistical independence? In a small amount of gas (almost by definition of what is a gas), the particles don't have much effect on each other. One can assume statistical independence.

While in space with gravity overwhelming other effects, the particles have very much effect on each other. Hence the statistics about their state are affected by these dependencies. So the previous intuition about entropy can't hold.

Post reply on HN