Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

johncarlosbaez.wordpress.com

Post reply on HN
Print view View original

201–210 of 221 posts

Re: What Is Entropy?

#201

Post by bubblyworld »

_hark wrote: post #182
bubblyworld wrote: post #158

Earlier quoted context omitted.

Right, this is exactly what I'm getting at - learning a distribution over a fixed sample space can be done with Bayesian methods, or entropy-based methods like the OP suggested, but I'm wondering if there are methods that can automatically adjust the sample space as well. For well-defined mathematical problems like dice rolling and fixed classical mechanics scenarios and such, you don't need this I guess, but for any…

We could discuss completeness of the sample space, and we can also discuss completeness of the hypothesis space . In Solomonoff Induction, which purports to be a theory of universal inductive inference, the "complete hypothesis space" consists of all computable programs (note that all current physical theories are computable, so this hypothesis space is very general). Then induction is performed by keeping all progra…

Someone else pointed me to Solomonoff induction too, which looks really cool as an "idealised" theory of induction and it definitely solves my question in abstract. But there are obvious difficulties with that in practice, like the fact that it's probably uncomputable, right?

I mean I think even the "Complexity" coefficient should be uncomputable in general, since you could probably use a program which computes it to upper bound "Complexity", and if there was such an upper bound you could use it to solve the halting problem etc. Haven't worked out the details though!

Would be interesting if there are practical algorithms for this. Either direct approximations to SI or maybe something else entirely that approaches SI in the limit, like a recursive neural-net training scheme? I'll do some digging, thanks!

Re: What Is Entropy?

#202

Post by kragen »

kragen wrote: post #190
marcosdumay wrote: post #186

Earlier quoted context omitted.

The entropy is objective if you completely define the communication channel, and subjective if you weave the definition away.

the subjectivity doesn't stem from the definition of the channel but from the model of the information source. what's the prior probability that you intended to say 'weave', for example? that depends on which model of your mind we are using. frequentists argue that there is an objectively correct model of your mind we should always use, and bayesians argue that it depends on our prior knowledge about your mind

(i mean, your information about what the channel does is also potentially incomplete, so the same divergence in definitions could arise there too, but the subjectivity doesn't just stem from the definition of the channel; and shannon entropy is a property that can be imputed to a source independent of any channel)

Re: What Is Entropy?

#203

Post by GoblinSlayer »

space_oddity wrote: post #141
GoblinSlayer wrote: post #107

There's fundamental nature of entropy, but as usual it's not very enlightening for poor monkey brain, so to explain you need to enumerate all its high level behavior, but its high level behavior is accidental and can't be summarized in a concise form.

This complexity underscores the richness of the concept

I'd say it underscores its accidental nature.

Re: What Is Entropy?

#204

Post by utkarsh858 »

empath75 wrote: post #164
utkarsh858 wrote: post #74

I sometimes ponder where new entropy/randomness is coming from, like if we take the earliest state of universe as an infinitely dense point particle which expanded. So there must be some randomness or say variety which led it to expand in a non uniform way which led to the dominance of matter over anti-matter, or creation of galaxies, clusters etc. If we take an isolated system in which certain static particles are p…

Symmetry breaking is the general phenomenon that underlies most of that. The classic example is this: Imagine you have a perfectly symmetrical sombrero[1], and there's a ball balanced on top of the middle of the hat. There's no preferred direction it should fall in, but it's _unstable_. Any perturbation will make it roll down hill and come to rest in a stable configuration on the brim of the hat. The symmetry of the…

Yes but what will intimate that perturbation?

Re: What Is Entropy?

#205

Post by utkarsh858 »

pseidemann wrote: post #140
utkarsh858 wrote: post #74

I sometimes ponder where new entropy/randomness is coming from, like if we take the earliest state of universe as an infinitely dense point particle which expanded. So there must be some randomness or say variety which led it to expand in a non uniform way which led to the dominance of matter over anti-matter, or creation of galaxies, clusters etc. If we take an isolated system in which certain static particles are p…

I saw this video, which explained it for me (it's german, maybe the automatic subtitles will work for you): https://www.youtube.com/watch?v=hrJViSH6Klo He argues that the randomness you are looking for comes from quantum fluctuations, and if this randomness did not exist, the universe would probably never have "happened".

Thanks for the reference will take some time before I see the whole video. Can you tell me what those quantum fluctuations are in short? Are they part of some physical law?

Re: What Is Entropy?

#206

Post by financltravsty »

mitthrowaway2 wrote: post #198
financltravsty wrote: post #192

Earlier quoted context omitted.

A representative sample then? Please tell me where I went wrong -- I mean this sincerely.

This might be a frequentist vs bayesian thing, and I am bayesian. So maybe other people would have a different view. I don't think you need to have any information to have a probability distribution; your distribution already represents your degree of ignorance about an outcome. So without even sampling it once, you already should have a uniform probability distribution for a random number generator or a coin flip. I…

I only slightly understand, I'm sorry; I'm not educated enough to understand much of this.

Did you take stats at MIT? I'm going to through their online material, because I very much am very confused.

Re: What Is Entropy?

#207

Post by Herodic »

cubefox wrote: post #184
bglazer wrote: post #168

Earlier quoted context omitted.

Emil Kirkegaard is a self-described white nationalist eugenicist who thinks the age of consent is too high. I wouldn't trust anything he has to say.

> Emil Kirkegaard is a self-described white nationalist That's simply a lie. > who thinks the age of consent is too high Too high in which country? Such laws vary strongly, even by US state, and he is from Denmark. Anyway, this has nothing to do with the topic at hand.

[flagged]

Re: What Is Entropy?

#208

Post by adrianN »

strogonoff wrote: post #200
Salgat wrote: post #170

Earlier quoted context omitted.

Centuries ago, the limitation of most knowledge was the difficulty in discovery; once known, it was accessible to most scholars. Take Calculus, which is taught in every high school in America. The problem is, we're getting to a point where new fields are built on such extreme requirements, that even the known knowledge is extremely hard for talented university students to learn, let alone what is required to discover…

Sounds like the increased difficulty could be addressed with new models and right abstraction layers. E.g., there’s incredible complexity in modern computing, but you don’t need to know assembly in order build a Web app, to reason about architecture, to operate functional paradigms, etc. However, this doesn’t seem to happen in natural sciences. I wonder if adopting better models runs into the gatekeepers protecting t…

Of course it happens in natural sciences. The neuroscientist doesn't need to to quantum mechanical calculations to do research.

Re: What Is Entropy?

#209

Post by Herodic »

Herodic wrote: post #207
cubefox wrote: post #184

Earlier quoted context omitted.

> Emil Kirkegaard is a self-described white nationalist That's simply a lie. > who thinks the age of consent is too high Too high in which country? Such laws vary strongly, even by US state, and he is from Denmark. Anyway, this has nothing to do with the topic at hand.

[flagged]

[flagged]

Re: What Is Entropy?

#210

Post by Herodic »

bglazer wrote: post #168
cubefox wrote: post #130

Earlier quoted context omitted.

https://emilkirkegaard.dk/en/2022/11/a-theory-of-ashkenazi-g...

Emil Kirkegaard is a self-described white nationalist eugenicist who thinks the age of consent is too high. I wouldn't trust anything he has to say.

[flagged]
Post reply on HN