131–140 of 221 posts
For information theory, I've always thought of entropy as follows: "If you had a really smart compression algorithm, how many bits would it take to accurately represent this file?" i.e., Highly repetitive inputs compress well because they don't have much entropy per bit. Modern compression algorithms are good enough on most data to be used as a reasonable approximation for the true entropy.
Earlier quoted context omitted.
Some probability distributions are objective. The probability that my random number generator gives me a certain number is given by a certain formula. Describing it with another distribution would be wrong. Another example, if you have an electron in a superposition of half spin-up and half spin-down, then the probability to measure up is objectively 50%. Another example, GPT-2 is a probability distribution on sequen…
Of those, the quantum superposition is the only one that has a chance at being considered objective, and it's still only "objective" in the sense that (as far as we know) your description provided as much information as anyone can possibly have about it, so nobody can have a more-informed opinion and all subjects agree. The others are both partial-information problems which are very sensitive to knowing certain hidde…
Earlier quoted context omitted.
Are there many von-Neumann-like multidisciplinaries nowadays? It feels like unless one is razor sharp fully into one field one is not to be treated seriously by those who made careers in it (and who have the last word on it).
There have been a very small number of thinkers as publicly accomplished as von Neumann ever. One other who comes to mind is Carl F. Gauss.
Earlier quoted context omitted.
He was really brilliant, made contributions all over the place in the math/physics/tech field, and had a sort of wild and quirky personality that people love telling stories about. A funny quote about him from a Edward “a guy with multiple equations named after him” Teller: > Edward Teller observed "von Neumann would carry on a conversation with my 3-year-old son, and the two of them would talk as equals, and I somet…
Are there many von-Neumann-like multidisciplinaries nowadays? It feels like unless one is razor sharp fully into one field one is not to be treated seriously by those who made careers in it (and who have the last word on it).
Earlier quoted context omitted.
Misconceptions about entropy are misconceptions about physics. You can’t dispell them focusing on the maths and ignoring the physics entirely - especially if you just write an equation without any conceptual discussion, not even mathematical.
I didn't say to only focus on the mathematics. Obviously wherever you apply the concept (and it's applied to much more than physics) there will be other sources of confusion. But just knowing that entropy is a property of a distribution, not a state, already helps clarify your thinking. For instance, you know that the question "what is the entropy of a broken egg?" is actually meaningless, because you haven't specifi…
Anyway, it seems that - like many others - I just misunderstood the “little need for all the mystery” remark.
Earlier quoted context omitted.
And this is what I prefer too, although with the clarification that its the number of ways that a system can be arranged without changing its macroscopic properties . Its, unfortunately, not very compatible with Shannon's usage in any but the shallowest sense, which is why it stays firmly in the land of physics.
Assuming each of the N microstates for a given macrostate are equally possible with probability p=1/N, the Shannon Entropy is -Σp.log(p) = -N.p.log(p)=-1.log(1/N)=log(N), which is the physics interpretation. In the continuous version, you would get log(V) where V is the volume in phase space occupied by the microstates for a given macrostate. Liouville's theorem that the volume is conserved in phase space implies tha…
> If our goal is to predict the future, it suffices to choose a distribution that is uniform in the Liouville measure given to us by classical mechanics (or its quantum analogue). If we want to reconstruct the past, in contrast, we need to conditionalize over trajectories that also started in a low-entropy past state — that the “Past Hypothesis” that is required to get stat mech off the ground in a world governed by time-symmetric fundamental laws.
https://www.preposterousuniverse.com/blog/2013/07/09/cosmolo...
I sometimes ponder where new entropy/randomness is coming from, like if we take the earliest state of universe as an infinitely dense point particle which expanded. So there must be some randomness or say variety which led it to expand in a non uniform way which led to the dominance of matter over anti-matter, or creation of galaxies, clusters etc. If we take an isolated system in which certain static particles are p…
He argues that the randomness you are looking for comes from quantum fluctuations, and if this randomness did not exist, the universe would probably never have "happened".