Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Entropy explained, with sheep (2016)

aatishb.com

Post reply on HN
Print view View original

41–50 of 107 posts

Re: Entropy explained, with sheep (2016)

#41

Post by isanengineer »

andrewflnr wrote: post #16

So one thing I've never understood is how you can "count" microstates, or bits required to describe them, when the relevant physical parameters all seem to be real numbers. For instance, a gas of N atoms is described by 6N real numbers (3d position and velocity) regardless of how hot it is. The article talks about quanta of energy, but that seems like a simplification at best: a given interaction might be quantized,…

Good point. I think the what's missing is how electron energy states work in quantum mechanics. The wikipedia paged (linked below) has a pretty good explanation:

"A quantum mechanical system or particle that is bound—that is, confined spatially—can only take on certain discrete values of energy, called energy levels. This contrasts with classical particles, which can have any amount of energy. The term is commonly used for the energy levels of electrons in atoms, ions, or molecules, which are bound by the electric field of the nucleus, but can also refer to energy levels of nuclei or vibrational or rotational energy levels in molecules. The energy spectrum of a system with such discrete energy levels is said to be quantized". (https://en.wikipedia.org/wiki/Energy_level)

To put things another way: while there could theoretically be infinite sizes of energy quanta, the permutations of energy states for matter are in fact discrete.

Disclaimer: I am an engineer, not a physicist.

Re: Entropy explained, with sheep (2016)

#42

Post by layer8 »

That still doesn’t answer the question how, if the laws of physics are time-symmetric, the universe as a whole can have a time-asymmetric evolution of entropy. I.e., if something forces entropy to increase in the long run, then that should hold in both directions of time. So what is it that causes entropy to only increase in the direction of the future, but not in the direction of the past, given that the laws pf physics do not distinguish between both directions?

Re: Entropy explained, with sheep (2016)

#43

Post by marcosdumay »

novaRom wrote: post #24
jbay808 wrote: post #13

Earlier quoted context omitted.

> The mere fact of that seems to contradict our current understanding of entropy What part of our understanding does it contradict? The second law of thermodynamics says that entropy increases with time; this seems entirely consistent with a low-entropy past. One explanation for why the universe began in a low entropy state is that that state has a very low description length. (This is a bit of a truism, since descri…

Your reasoning is strange. Actually, higher entropy is what we may call "of lower complexity" requiring ever-shorter description length.

If you care about describing the details, you can compress your description better if it's a low-entropy state.

But of course, cosmology is full of more mundane explanations about how the limit of the possible entropy of the universe can grow with time, so a high-entropy state suddenly has a lot of room to increase even further.

Re: Entropy explained, with sheep (2016)

#44

Post by pabo »

maxov wrote: post #12

Loved this article. There are so many applications of entropy and statistical physics in computer science, and I find it fascinating that the same general properties are useful in such different contexts. For example, there's a well-known phenomenon in probability called concentration of measure. One of the most important examples in computer science is if you flip n coins independently, then the number of heads conc…

There's a related anecdote about John von Neumann: he used to joke that he has superpowers and can easily tell truly random and pseudo random sequences apart. He asked people to sit down in another room and generate a 0/1 sequence via coin flips, and record it. Then, generate another sequence by heart, trying to mimick randomness as much as possible. When people finally showed the two sequences to him, Neumann could instantly declare which one was which.

People were amazed.

The trick he used was based on the "burstiness" rule you describe: a long enough random sequence will likely contain a long homogeneous block. While humans tend to avoid long streaks of the same digit, as it does not feel random enough.

So, all he did was he quickly checked with a glimpse, which of the two sequences contained the longest homogeneous block, and recognized that as the one generated via the coin flips.

Re: Entropy explained, with sheep (2016)

#45

Post by marcosdumay »

keenmaster wrote: post #5

Great explanation. What are the top theories for why the universe began in a low entropy state? The mere fact of that seems to contradict our current understanding of entropy. That implies that there is something very fundamental about the universe which we don't understand.

Cosmology works a lot around that question. There are a few theories that claim that the maximum entropy of the universe is growing with time, so what was high-entropy on the past gets room to keep growing.

Re: Entropy explained, with sheep (2016)

#46

Post by kergonath »

andrewflnr wrote: post #16

So one thing I've never understood is how you can "count" microstates, or bits required to describe them, when the relevant physical parameters all seem to be real numbers. For instance, a gas of N atoms is described by 6N real numbers (3d position and velocity) regardless of how hot it is. The article talks about quanta of energy, but that seems like a simplification at best: a given interaction might be quantized,…

This is a very good question, which goes to the heart of statistical Physics. We use phase spaces for this (typically a 6N-dimensional vector space in which each microstate is represented by a point). The system has a probability of being in (or rather very close to) each microstate, which depends on several factors, like the conditions (isolated system, constant pressure, temperature, number of particles, etc). Counting microstates is “just” calculating integrals of that probability weight in the phase space. Of course, most of the time it is impossible, so we have tools to approximate these integrals. There are a lot of subtleties, but that’s the general idea.

The phase space does not change depending on temperature, so there’s nothing weird like the space getting bigger. But the probability of each microstate might, as high-energy states become more accessible.

Re: Entropy explained, with sheep (2016)

#47

Post by bobsoap »

I love how the notion of entropy permeates into so many other things. It's fundamental, universal, and at the heart of nearly every aspect of our existence.

Take philosophy. If the ultimate state of everything culminates in chaos (according to the theory of entropy), the human existence constitues the exact opposite: controlling the chaos that surrounds us, and shaping it into something useful and, in entropy-speak, progressively unprobable. Making ice cubes out of water.

It follows that our existence can at least be described as a function of entropy.

This describes life. Survival is the battle against entropy. Procreation is the chosen weapon against chaos, a force of order in a universe that can't help itself but fall into chaos -- and, undoubtedly, will ultimately prevail in that fight. In the long run, life will lose.

Great article. Fascinating stuff.

Re: Entropy explained, with sheep (2016)

#48

Post by antonios »

bobsoap wrote: post #47

I love how the notion of entropy permeates into so many other things. It's fundamental, universal, and at the heart of nearly every aspect of our existence. Take philosophy. If the ultimate state of everything culminates in chaos (according to the theory of entropy), the human existence constitues the exact opposite: controlling the chaos that surrounds us, and shaping it into something useful and, in entropy-speak,…

Alternatively, all that attempts to control chaos and decrease entropy actually results in faster entropy increase overall on a systemic level. I remember reading about a (Russian?) physicist that believed that life simply happens as a result of the universe's attempt to increase entropy faster on sufficiently complicated systems. If someone remembers his name I'd be obliged.

Re: Entropy explained, with sheep (2016)

#49

Post by kergonath »

isanengineer wrote: post #41
andrewflnr wrote: post #16

So one thing I've never understood is how you can "count" microstates, or bits required to describe them, when the relevant physical parameters all seem to be real numbers. For instance, a gas of N atoms is described by 6N real numbers (3d position and velocity) regardless of how hot it is. The article talks about quanta of energy, but that seems like a simplification at best: a given interaction might be quantized,…

Good point. I think the what's missing is how electron energy states work in quantum mechanics. The wikipedia paged (linked below) has a pretty good explanation: "A quantum mechanical system or particle that is bound—that is, confined spatially—can only take on certain discrete values of energy, called energy levels. This contrasts with classical particles, which can have any amount of energy. The term is commonly us…

What you say is true, but also incomplete. We are perfectly able to quantify the accessible states in purely classical systems, such as ideal gases, without requiring discrete energy levels. The trick is to think of a continuous probability density instead of discrete probabilities. This framework is very general and does not depend on the quantum-ness of what you look at.

Even in some systems that actually follow quantum mechanics (such as phonons or electrons in a material, or photons in a black body), we often use continuous probabilities (densities of states) because it’s much more convenient when you have lots of particles.

Re: Entropy explained, with sheep (2016)

#50

Post by kergonath »

foota wrote: post #27
andrewflnr wrote: post #16

So one thing I've never understood is how you can "count" microstates, or bits required to describe them, when the relevant physical parameters all seem to be real numbers. For instance, a gas of N atoms is described by 6N real numbers (3d position and velocity) regardless of how hot it is. The article talks about quanta of energy, but that seems like a simplification at best: a given interaction might be quantized,…

IANAP but there's two different things that come to mind. Even if the number of states were infinite, as long as there's a reasonable probability distribution you can know you can integrate over the probabilities that have some property vs another (say: solid vs not). Secondly, energy is quantized (on a very very very small level) in reality. Ymmv though, I tried googling this and read some stuff about waves being qu…

For quantum objects, the distinction between a wave and a particle is not very meaningful. The energy levels of an electron around a nucleus are discrete, regardless of whether the electron behaves more like a wave or more like a particle in the specific experiment you’re doing.

Otherwise, you’re right: we count states by integrating (at times discrete, continuous, and often very complex) probability distributions.

Post reply on HN