[flagged]
What Is Entropy?
211–220 of 221 posts
Re: What Is Entropy?
#212Earlier quoted context omitted.
> Which we don’t know precisely. Entropy is about not knowing. No, it is not about not knowing. This is an instance of the intuition from Shannon’s entropy does not translate to statistical Physics. It is about the number of possible microstates, which is completely different. In Physics, entropy is a property of a bit of matter, it is not related to the observer or their knowledge. We can measure the enthalpy change…
>>> Particles have a defined position and momentum [...] If you somehow learned these then the shannon entropy is zero. >> Entropy in classical statistical mechanics is proportional to the logarithm of the volume in phase space [and diverges to minus infinity if you define precisely the position and momentum of the particles and the volume in phase sphere goes to zero] > [It's zero also] if you consider the phase spa…
I hadn't realized that "differential" entropy and shannon entropy are actually different and incompatible, huh.
So the case I mentioned, where you know all the positions and momentums has 0 shannon entropy and -Inf differential entropy. And a typical distribution will instead have Inf shannon entropy and finite differential entropy.
Wikipedia has some pretty interesting discussion about Differential Entropy vs Limiting density of Points, but I can't claim to understand it and whether it could bridge the gap here.
Re: What Is Entropy?
#213Earlier quoted context omitted.
This might be a frequentist vs bayesian thing, and I am bayesian. So maybe other people would have a different view. I don't think you need to have any information to have a probability distribution; your distribution already represents your degree of ignorance about an outcome. So without even sampling it once, you already should have a uniform probability distribution for a random number generator or a coin flip. I…
I only slightly understand, I'm sorry; I'm not educated enough to understand much of this. Did you take stats at MIT? I'm going to through their online material, because I very much am very confused.
I appreciate your curiosity!
The best introduction that I can recommend is this type-written PDF from E.T. Jaynes, called "probability theory with applications in science and engineering": https://bayes.wustl.edu/etj/science.pdf.html
It requires a lot of attention to read and follow the math, but it's worthwhile. Jaynes is a pretty passionate writer, and in his writing he's clearly battling against some enemies (who might be ghosts), but on the other hand this also makes for more entertaining reading and I find that's usually a benefit when it comes to a textbook.
Re: What Is Entropy?
#214Earlier quoted context omitted.
Sounds like the increased difficulty could be addressed with new models and right abstraction layers. E.g., there’s incredible complexity in modern computing, but you don’t need to know assembly in order build a Web app, to reason about architecture, to operate functional paradigms, etc. However, this doesn’t seem to happen in natural sciences. I wonder if adopting better models runs into the gatekeepers protecting t…
Of course it happens in natural sciences. The neuroscientist doesn't need to to quantum mechanical calculations to do research.
Neither does a Web app developer need to know how to use CNC or make a transistor. Your example is about different levels of abstraction than what I meant.
I was replying to “even the known knowledge is extremely hard for talented university students to learn”. If complexity of the known knowledge one must learn to substantially contribute is the reason becoming an accomplished multidisciplinary is impossible nowadays, then it sounds like we could use some better models and levels of abstraction.
Re: What Is Entropy?
#215Earlier quoted context omitted.
>>> Particles have a defined position and momentum [...] If you somehow learned these then the shannon entropy is zero. >> Entropy in classical statistical mechanics is proportional to the logarithm of the volume in phase space [and diverges to minus infinity if you define precisely the position and momentum of the particles and the volume in phase sphere goes to zero] > [It's zero also] if you consider the phase spa…
I hadn't realized that "differential" entropy and shannon entropy are actually different and incompatible, huh . So the case I mentioned, where you know all the positions and momentums has 0 shannon entropy and -Inf differential entropy. And a typical distribution will instead have Inf shannon entropy and finite differential entropy. Wikipedia has some pretty interesting discussion about Differential Entropy vs Limit…
> So the case I mentioned, where you know all the positions and momentums has 0 shannon entropy
No, Shannon entropy is not applicable in that case.
https://en.wikipedia.org/wiki/Entropy_(statistical_thermodyn...
Quantum mechanics solves the issue of the continuity of the state space. However, as you probably know, in quantum mechanics all the positions and momentums cannot simultaneously have definite values.
Re: What Is Entropy?
#216Earlier quoted context omitted.
Would you say that all claims about the world are subjective, because they have to be based on someone’s observations? For example my cat weighs 13 pounds. That seems objective, in the sense that if two people disagree, only one can be right. But the claim is based on my observations. I think your logic leads us to deny that anything is objective.
I do believe in objective reality, but probabilities are subjective. Your cat weighs 13 pounds, and now that you've told me, I know it too. If you asked me to draw a probability distribution for the weight of your cat, I'd draw a tight gaussian distribution around that, representing the accuracy of your scale. My cat weighs a different amount, but I won't tell you how much, so if we both draw a probability distributi…
All right now, here's the big question:
how do you know that the evidence your
sensory apparatus reveals to you is correct?
What I'm getting at is this: the only
experience that is directly available to
you is your sensory data. And this sensory
data is merely a stream of electrical
impulses which stimulate your computing
center.
In other words, all that I really know
about the outside universe is relayed to
me through my electrical connections.
Why, that would mean that...I really don't
know what the outside universe is like at
all, for certain.Re: What Is Entropy?
#217Earlier quoted context omitted.
I only slightly understand, I'm sorry; I'm not educated enough to understand much of this. Did you take stats at MIT? I'm going to through their online material, because I very much am very confused.
I appreciate your curiosity! The best introduction that I can recommend is this type-written PDF from E.T. Jaynes, called "probability theory with applications in science and engineering": https://bayes.wustl.edu/etj/science.pdf.html It requires a lot of attention to read and follow the math, but it's worthwhile. Jaynes is a pretty passionate writer, and in his writing he's clearly battling against some enemies (who…
I read through the first "lecture" yesterday. I'll devote some time for (hopefully) the rest today.
Thank you!
Re: What Is Entropy?
#218Earlier quoted context omitted.
There's a good case to be made that the information-theoretic definition of entropy is the most fundamental one, and the version that shows up in physics is just that concept as applied to physics.
Yeah, people seemingly misunderstand that the entropy applied to thermodynamics is simply an aggregate statistic that summarizes the complex state of the thermodynamic system as a single real number. The fact that entropy always rises etc, has nothing to do with the statistical concept of entropy itself. It simply is an easier way to express the physics concept that individual atoms spread out their kinetic energy ac…
I'm not sure that's quite the right perspective. It's not a coincidence that entropy increases over time; the increase in entropy seems to be very fundamental to the way physics goes. I prefer the interpretation "physics doesn't care what direction the arrow of time points, but we perceive it as pointing in the direction of increasing entropy". Although that's not totally satisfying either.
Re: What Is Entropy?
#219Earlier quoted context omitted.
> Which we don’t know precisely. Entropy is about not knowing. No, it is not about not knowing. This is an instance of the intuition from Shannon’s entropy does not translate to statistical Physics. It is about the number of possible microstates, which is completely different. In Physics, entropy is a property of a bit of matter, it is not related to the observer or their knowledge. We can measure the enthalpy change…
> In Physics, entropy is a property of a bit of matter, it is not related to the observer or their knowledge. We can measure the enthalpy change of a material sample and work out its entropy without knowing a thing about its structure. Enthalpy is also dependent on your choice of state variables, which is in turn dictated by which observables you want to make predictions about: whether two microstates are distinguish…
A calorimeter does not care about anyone’s choice of state variables. Entropy is not only something that exists in abstract theoretical constructs, it is something we can get experimentally.
Re: What Is Entropy?
#220A well known anecdote reported by Shannon: "My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that…
Von Neumann was the king of kings
I disagree... Von Neumann went beyond being a King of Kings, the man was a God (or a "Monster Mind" according to Feynman) :)