Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

johncarlosbaez.wordpress.com

Post reply on HN
Print view View original

161–170 of 221 posts

Re: What Is Entropy?

#161

Post by [deleted] »

financltravsty wrote: post #142
canjobear wrote: post #71

Earlier quoted context omitted.

Some probability distributions are objective. The probability that my random number generator gives me a certain number is given by a certain formula. Describing it with another distribution would be wrong. Another example, if you have an electron in a superposition of half spin-up and half spin-down, then the probability to measure up is objectively 50%. Another example, GPT-2 is a probability distribution on sequen…

The probability distribution is subjective for both parts -- because it, once again, depends on the observer observing the events in order to build a probability distribution. E.g. your random number generator generates 1, 5, 7, 8, 3 when you run it. It generates 4, 8, 8, 2, 5 when I run it. I.e. we have received different information about the random number generator to build our subjective probability distributions…

[deleted]

Re: What Is Entropy?

#162

Post by ricksunny »

adrianN wrote: post #144
bee_rider wrote: post #136

Earlier quoted context omitted.

I think there are none. The world has gotten too complicated for that. It was early days in quantum physics, information theory, and computer science. I don’t think it is early days in anything that consequential anymore.

It’s the early days in a lot of fields, but they tend to be fiendishly difficult like molecular biology or neuroscience.

More than that, as professionals' career paths in fields develop, the organisations they work for specialize, becoming less amenable to the generalist. ('Why should we hire this mathematician who is also an expert in legal research? Their attention is probably divided, and meanwhile we have a 100% mathematician in the candidate pool fresh from an expensive dedicated PhD program with a growing family to feed.')

I'm obviously using the archetype of Leibniz here as an example but pick your favorite polymath.

Re: What Is Entropy?

#163

Post by complaintdept »

tonetegeatinst wrote: post #61
BigParm wrote: post #13

Earlier quoted context omitted.

Von Neumann was the king of kings

Its odd...as someone interested but not fully into the sciences I see his name pop up everywhere.

Even mortals such as ourselves can apply some of Von Neumann's ideas in our everyday lives:

https://en.m.wikipedia.org/wiki/Fair_coin#Fair_results_from_...

Re: What Is Entropy?

#164

Post by empath75 »

utkarsh858 wrote: post #74

I sometimes ponder where new entropy/randomness is coming from, like if we take the earliest state of universe as an infinitely dense point particle which expanded. So there must be some randomness or say variety which led it to expand in a non uniform way which led to the dominance of matter over anti-matter, or creation of galaxies, clusters etc. If we take an isolated system in which certain static particles are p…

Symmetry breaking is the general phenomenon that underlies most of that.

The classic example is this:

Imagine you have a perfectly symmetrical sombrero[1], and there's a ball balanced on top of the middle of the hat. There's no preferred direction it should fall in, but it's _unstable_. Any perturbation will make it roll down hill and come to rest in a stable configuration on the brim of the hat. The symmetry of the original configuration is now broken, but it's stable.

1: https://m.media-amazon.com/images/I/61M0LFKjI9L.__AC_SX300_S...

Re: What Is Entropy?

#165

Post by canjobear »

financltravsty wrote: post #142
canjobear wrote: post #71

Earlier quoted context omitted.

Some probability distributions are objective. The probability that my random number generator gives me a certain number is given by a certain formula. Describing it with another distribution would be wrong. Another example, if you have an electron in a superposition of half spin-up and half spin-down, then the probability to measure up is objectively 50%. Another example, GPT-2 is a probability distribution on sequen…

The probability distribution is subjective for both parts -- because it, once again, depends on the observer observing the events in order to build a probability distribution. E.g. your random number generator generates 1, 5, 7, 8, 3 when you run it. It generates 4, 8, 8, 2, 5 when I run it. I.e. we have received different information about the random number generator to build our subjective probability distributions…

Would you say that all claims about the world are subjective, because they have to be based on someone’s observations?

For example my cat weighs 13 pounds. That seems objective, in the sense that if two people disagree, only one can be right. But the claim is based on my observations. I think your logic leads us to deny that anything is objective.

Re: What Is Entropy?

#166

Post by canjobear »

mitthrowaway2 wrote: post #90
canjobear wrote: post #71

Earlier quoted context omitted.

Some probability distributions are objective. The probability that my random number generator gives me a certain number is given by a certain formula. Describing it with another distribution would be wrong. Another example, if you have an electron in a superposition of half spin-up and half spin-down, then the probability to measure up is objectively 50%. Another example, GPT-2 is a probability distribution on sequen…

Of those, the quantum superposition is the only one that has a chance at being considered objective, and it's still only "objective" in the sense that (as far as we know) your description provided as much information as anyone can possibly have about it, so nobody can have a more-informed opinion and all subjects agree. The others are both partial-information problems which are very sensitive to knowing certain hidde…

You might have some probability distribution in your head for what will come out of GPT-2 on your machine at a certain time, based on your knowledge of the random seed. But that is not the GPT-2 probability distribution, which is objectively defined by model weights that you can download, and which does not correspond to anyone’s beliefs.

Re: What Is Entropy?

#167

Post by bee_rider »

adrianN wrote: post #144
bee_rider wrote: post #136

Earlier quoted context omitted.

I think there are none. The world has gotten too complicated for that. It was early days in quantum physics, information theory, and computer science. I don’t think it is early days in anything that consequential anymore.

It’s the early days in a lot of fields, but they tend to be fiendishly difficult like molecular biology or neuroscience.

Are they fiendishly difficult or do we just need a von Neumann to come along and do what he did for quantum mechanics to them?

Re: What Is Entropy?

#168

Post by bglazer »

cubefox wrote: post #130
zeristor wrote: post #121

Earlier quoted context omitted.

I was hoping the Wikipedia might explain why this might have been.

https://emilkirkegaard.dk/en/2022/11/a-theory-of-ashkenazi-g...

Emil Kirkegaard is a self-described white nationalist eugenicist who thinks the age of consent is too high. I wouldn't trust anything he has to say.

Re: What Is Entropy?

#169

Post by canjobear »

bubblyworld wrote: post #104
canjobear wrote: post #78

Earlier quoted context omitted.

What's often lost in the discussions about whether entropy is subjective or objective is that, if you dig a little deeper, information theory gives you powerful tools for relating the objective and the subjective. Consider cross entropy of two distributions H[p, q] = -Σ p_i log q_i. For example maybe p is the real frequency distribution over outcomes from rolling some dice, and q is your belief distribution. You can…

Thanks, that's an interesting perspective. It also highlights one of the weak points in the concept, I think, which is that this is only a tool for updating beliefs to the extent that the underlying probability space ("ontology" in this analogy) can actually "model" the phenomenon correctly! It doesn't seem to shed much light on when or how you could update the underlying probability space itself (or when to change y…

This kind of thinking will lead you to ideas like algorithmic probability, where distributions are defined using universal Turing machines that could model anything.

Re: What Is Entropy?

#170

Post by Salgat »

adrianN wrote: post #144
bee_rider wrote: post #136

Earlier quoted context omitted.

I think there are none. The world has gotten too complicated for that. It was early days in quantum physics, information theory, and computer science. I don’t think it is early days in anything that consequential anymore.

It’s the early days in a lot of fields, but they tend to be fiendishly difficult like molecular biology or neuroscience.

Centuries ago, the limitation of most knowledge was the difficulty in discovery; once known, it was accessible to most scholars. Take Calculus, which is taught in every high school in America. The problem is, we're getting to a point where new fields are built on such extreme requirements, that even the known knowledge is extremely hard for talented university students to learn, let alone what is required to discover and advance that field. Until we are able to augment human intelligence, the days of the polymath advancing multiple fields are mostly over. I would also argue that the standards for peer reviewed whitepapers and obtaining PhDs has significantly dropped (due to the incentive structure to spam as many papers as possible), which is only hurting the advancement of knowledge.
Post reply on HN