Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

johncarlosbaez.wordpress.com

Post reply on HN
Print view View original

171–180 of 221 posts

Re: What Is Entropy?

#171

Post by nyssos »

kergonath wrote: post #156
kgwgk wrote: post #153

Earlier quoted context omitted.

> Particles have a defined position and momentum Which we don’t know precisely. Entropy is about not knowing. > If you somehow learned these then the shannon entropy is zero. Minus infinity. Entropy in classical statistical mechanics is proportional to the logarithm of the volume in phase space. (You need an appropriate extension of Shannon’s entropy to continuous distributions.) > So now you are forced to consider e…

> Which we don’t know precisely. Entropy is about not knowing. No, it is not about not knowing. This is an instance of the intuition from Shannon’s entropy does not translate to statistical Physics. It is about the number of possible microstates, which is completely different. In Physics, entropy is a property of a bit of matter, it is not related to the observer or their knowledge. We can measure the enthalpy change…

> In Physics, entropy is a property of a bit of matter, it is not related to the observer or their knowledge. We can measure the enthalpy change of a material sample and work out its entropy without knowing a thing about its structure.

Enthalpy is also dependent on your choice of state variables, which is in turn dictated by which observables you want to make predictions about: whether two microstates are distinguishable, and thus whether the part of the same macrostate, depends on the tools you have for distinguishing them.

Re: What Is Entropy?

#172

Post by enugu »

im3w1l wrote: post #151
enugu wrote: post #119

Earlier quoted context omitted.

Don't see how the connection is superficial. Even the classical macroscopic definition of entropy as ΔS=∫TdQ can be derived from the information theory perspective as Baez shows in article(using entropy maximizing distributions and Lagrange multipliers). If you have a more specific critique, it would be good to discuss.

In classical physics there is no real objective randomness. Particles have a defined position and momentum and those evolve deterministically. If you somehow learned these then the shannon entropy is zero. If entropy is zero then all kinds of things break down. So now you are forced to consider e.g. temperature an impossibility without quantum-derived randomness, even though temperature does not really seem to be a q…

> If entropy is zero then all kinds of things break down.

Entropy is a macroscopic variable and if you allow microscopic information, strange things can happen! One can move from a high entropy macrostate to a low entropy macrostate if you choose the initial microstate carefully. But this is not a reliable process which you can reproduce experimentally, ie. it is not a thermodynamic process.

A thermodynamics process P is something which takes a macrostate A to a macrostate B, independent of which microstate a0, a1, a2.. in A you started off with it. If the process depends on microstate, then it wouldn't be something we would recognize as we are looking from the macro perspective.

Re: What Is Entropy?

#173

Post by djd3 »

i_am_proteus wrote: post #125
strogonoff wrote: post #120

Earlier quoted context omitted.

Are there many von-Neumann-like multidisciplinaries nowadays? It feels like unless one is razor sharp fully into one field one is not to be treated seriously by those who made careers in it (and who have the last word on it).

There have been a very small number of thinkers as publicly accomplished as von Neumann ever. One other who comes to mind is Carl F. Gauss.

Euler.

JVM was one of the smartest ever, but Euler was there centuries before and shows up in so many places.

If I had a Time Machine I'd love to get those two together for a stiff drink and a banter.

Re: What Is Entropy?

#174

Post by suoduandao3 »

I like the formulation of 'the amount of information we don't know about a system that we could in theory learn'. I'm surprised there's no mention of the Copenhagen interpretation's interaction with this definition, under a lot of QM theories 'unavailable information' is different from available information.

Re: What Is Entropy?

#175

Post by YeGoblynQueenne »

bglazer wrote: post #168
cubefox wrote: post #130

Earlier quoted context omitted.

https://emilkirkegaard.dk/en/2022/11/a-theory-of-ashkenazi-g...

Emil Kirkegaard is a self-described white nationalist eugenicist who thinks the age of consent is too high. I wouldn't trust anything he has to say.

No need for ad hominems. This suffices to place doubt on the article's premises (and therefore any conclusion):

>> This hasn’t been strictly shown mathematically, but I think it is true.

Re: What Is Entropy?

#176

Post by tsoukase »

After years of thought I dare to say the 2nd TL is a tautology. Entropy is increasing means every system tends to higher probability means the most probable is the most probable.

Re: What Is Entropy?

#177

Post by passion__desire »

i_am_proteus wrote: post #125
strogonoff wrote: post #120

Earlier quoted context omitted.

Are there many von-Neumann-like multidisciplinaries nowadays? It feels like unless one is razor sharp fully into one field one is not to be treated seriously by those who made careers in it (and who have the last word on it).

There have been a very small number of thinkers as publicly accomplished as von Neumann ever. One other who comes to mind is Carl F. Gauss.

Genius Edward Teller Describes 1950s Genius John Von Neumann

https://youtu.be/Oh31I1F2vds?t=189 Describes Von Neumann's final days struggle when he couldn't think. Thinking, an activity which he loved the most.

Re: What Is Entropy?

#178

Post by bubblyworld »

canjobear wrote: post #169
bubblyworld wrote: post #104

Earlier quoted context omitted.

Thanks, that's an interesting perspective. It also highlights one of the weak points in the concept, I think, which is that this is only a tool for updating beliefs to the extent that the underlying probability space ("ontology" in this analogy) can actually "model" the phenomenon correctly! It doesn't seem to shed much light on when or how you could update the underlying probability space itself (or when to change y…

This kind of thinking will lead you to ideas like algorithmic probability, where distributions are defined using universal Turing machines that could model anything.

Amazing! I had actually heard about solomonoff induction before but my brain didn't make the connection. Thanks for the shortcut =)

Re: What Is Entropy?

#179

Post by tel »

bubblyworld wrote: post #104
canjobear wrote: post #78

Earlier quoted context omitted.

What's often lost in the discussions about whether entropy is subjective or objective is that, if you dig a little deeper, information theory gives you powerful tools for relating the objective and the subjective. Consider cross entropy of two distributions H[p, q] = -Σ p_i log q_i. For example maybe p is the real frequency distribution over outcomes from rolling some dice, and q is your belief distribution. You can…

Thanks, that's an interesting perspective. It also highlights one of the weak points in the concept, I think, which is that this is only a tool for updating beliefs to the extent that the underlying probability space ("ontology" in this analogy) can actually "model" the phenomenon correctly! It doesn't seem to shed much light on when or how you could update the underlying probability space itself (or when to change y…

You can sort of do this over a suitably large (or infinite) family of models all mixed, but from an epistemological POV that’s pretty unsatisfying.

From a practical POV it’s pretty useful and common (if you allow it to describe non- and semi-parametric models too).

Re: What Is Entropy?

#180

Post by tel »

tsoukase wrote: post #176

After years of thought I dare to say the 2nd TL is a tautology. Entropy is increasing means every system tends to higher probability means the most probable is the most probable.

I think that’s right, though it’s non-obvious that more probable systems are disordered. At least as non-obvious as Pascal’s triangle is.

Which is to say, worth saying from a first principles POV, but not all that startling.

Post reply on HN