Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It took me 10 years to understand entropy

cantorsparadise.com

Post reply on HN
Print view View original

81–90 of 295 posts

Re: It took me 10 years to understand entropy

#81

Post by andi999 »

jbay808 wrote: post #73
threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a ther…

Entropy (differences) are an objective quantity which can be measured, there is no subjectivity about it. It is not which parameters you know it is about which parameters you hold fixed.

Re: It took me 10 years to understand entropy

#82

Post by vishnugupta »

jbay808 wrote: post #73
threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a ther…

This is how I have come to understand entropy. The words disorder and order are a proxy for information content.

> If you know more about the system, it has less entropy.

One question though. When you say "it" does it include you as well as the system or just the system? To me "it" includes both because by it is "you" who's state has changed by acquiring more information. It could be in the form of neuronal rearrangement or bits being stored in some digital media etc., A new information content has thus been created.

There's an interesting side effect if one thinks deep enough here. The system will keep changing its state so the information one is out of date thus leading to more disorder (i.e., information loss) and increased entropy. One can keep the information updated but it takes energy. And I read somewhere that the energy thus used will lead to increase in overall entropy of the universe and thus the 2nd law.

Re: It took me 10 years to understand entropy

#83

Post by prof-dr-ir »

ogogmad wrote: post #59

I thought entropy (in the Shannon sense) was a property of discrete and finite probability distributions. It's essentially a measure of how random a sample from such a probability distribution is. Notably, continuous probability distributions don't have meaningful entropy (or in some sense, their entropy is always infinite). It's worth considering the similarities and differences between entropy and standard deviatio…

I think this comment is significantly more insightful than the article.

As for the thing you don't get: quantum mechanics means that the state space is actually discrete, which means there is no need to pass to a continuous distribution. And finiteness is not really a concern either: first of all it is not strictly necessary for the (Gibbs) entropy to be defined, and secondly the space state is actually often finite once e.g. the total energy in the system is fixed.

Re: It took me 10 years to understand entropy

#84

Post by nathias »

evouga wrote: post #25

One aspect of entropy that I always find counterintuitive is that unlike mass, charge, etc. it is not a physical quantity. In fact, from the point of view of an experimenter with perfect information about a physical system, the entropy of the system is exactly conserved over time (as made precise by Liouville's Theorem). The Second Law survives in this setting only in the most trivial sense that a constant function d…

It's a property of information so if you assume perfect information, of course it becomes trivial.

Re: It took me 10 years to understand entropy

#85

Post by dr_dshiv »

ogogmad wrote: post #59

I thought entropy (in the Shannon sense) was a property of discrete and finite probability distributions. It's essentially a measure of how random a sample from such a probability distribution is. Notably, continuous probability distributions don't have meaningful entropy (or in some sense, their entropy is always infinite). It's worth considering the similarities and differences between entropy and standard deviatio…

Right, it requires a sort of alphabet of discrete specific states. Discrete locations in space, discrete numbers of things and discrete kinds of things.

Re: It took me 10 years to understand entropy

#86

Post by dr_dshiv »

javajosh wrote: post #23
jleyank wrote: post #7

Earlier quoted context omitted.

Nah, Ultraviolet Catastrophe is worse, then there's wavefunction collapse. And there's gotta be something in particle physics that puts these terms to shame. Given that the particle names are drawn from literature and whimsy. I looked up the dictionary definition of pretentious, and so it doesn't really apply, but the hERG channel is a critical ion channel and various drugs block this and cause Really Bad Things. hER…

"Ultraviolet Catastrophe"'s problem is that it sounds way cooler than it is. I mean, that's the title of a cyberpunk novel; more hyperbolic (and disappointing) than pretentious. Eigenthings would be second on my list of pretentiousness (oddly, not gedankenthings; I'm inconsistent I guess). Standard Model names strike me whimsical to the point of being undignified.

Ultraviolet catastrophe is cool! It’s why we needed quantum physics, otherwise there would be catastrophic infinite energy emerging from random high frequency oscillations!

Re: It took me 10 years to understand entropy

#87

Post by a-dub »

ogogmad wrote: post #59

I thought entropy (in the Shannon sense) was a property of discrete and finite probability distributions. It's essentially a measure of how random a sample from such a probability distribution is. Notably, continuous probability distributions don't have meaningful entropy (or in some sense, their entropy is always infinite). It's worth considering the similarities and differences between entropy and standard deviatio…

yeah. i think of minimum entropy as a dirac delta distribution and maximum entropy as a flat uniform random distribution.

i never really understood the physical definition, but always handwaved it away with "things dissipate over time into an undetectable signal, or a flat distribution"

Re: It took me 10 years to understand entropy

#88

Post by mhh__ »

l33t2328 wrote: post #48
chemmail wrote: post #42

I had an art teacher who was very philosophical. One day he described to the class what entropy was. I took a lot of physics and even astrophysics. Little did i know he had a better conceptual understanding and explanation than i've ever heard before. Too bad i don't remember exactly what he said.

Not to poke holes in your nostalgia, but how do you know he had a great explanation if you don’t remember it after further study?

"Don't try to understand it, feel it" - from tenet, but does sort of apply to entropy as a way of looking at problems.

That being said I heard a sports science student try to recall their working definition of entry and it was some mess of locks and keys floating around randomly hitting eachother?

Re: It took me 10 years to understand entropy

#89

Post by dr_dshiv »

7speter wrote: post #17

Of all places, it was a conversation about the Socratic Forms in a Political Theory course I took in college that really brought the weight of the concept home to me. It went something like "Unlike the realm Socratic forms exist in, everything in our universe is subject to entropy; it is in everything's nature to degrade or decay over time." Maybe there's more to that? I'm all ears.

The forms are a low energy state that materially arise from an entropic process. Degradation and decay can lead to more organization and beauty.

Re: It took me 10 years to understand entropy

#90

Post by empiricus »

My understanding of entropy: it is a measure of how big a system (matter + energy from a space region) is, and how much its components have interacted with each other: Entropy ~ log(number of possible system states). As the universe unfolds, systems originally isolated are starting to interact and to form bigger systems, hence the number of possible states increases, and entropy increases too.
Post reply on HN