Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

jasonfantl.com

Post reply on HN
Print view View original

21–30 of 123 posts

Re: What Is Entropy?

#21

Post by glial »

empath75 wrote: post #17
glial wrote: post #8

One thing that helped me was the realization that, at least as used in the context of information theory, entropy is a property of an individual (typically the person receiving a message) and NOT purely of the system or message itself. > entropy quantifies uncertainty This sums it up. Uncertainty is the property of a person and not a system/message. That uncertainty is a function of both a person's model of a system/…

> If we're calculating the entropy of dice rolls (where the outcome is the 'message'), and I know the dice are loaded but you don't, my entropy will be lower than yours. That's got nothing to do with entropy being subjective. If 2 people are calculating any property and one of them is making a false assumption, they'll end up with a different (false) conclusion.

Entropy is based on your model of the world and every model, being a simplification and an estimate, is false.

Re: What Is Entropy?

#22

Post by mitthrowaway2 »

empath75 wrote: post #17
glial wrote: post #8

One thing that helped me was the realization that, at least as used in the context of information theory, entropy is a property of an individual (typically the person receiving a message) and NOT purely of the system or message itself. > entropy quantifies uncertainty This sums it up. Uncertainty is the property of a person and not a system/message. That uncertainty is a function of both a person's model of a system/…

> If we're calculating the entropy of dice rolls (where the outcome is the 'message'), and I know the dice are loaded but you don't, my entropy will be lower than yours. That's got nothing to do with entropy being subjective. If 2 people are calculating any property and one of them is making a false assumption, they'll end up with a different (false) conclusion.

What if I told you the dice were loaded, but I didn't tell you which face they were loaded in favor of?

Then you (presumably) assign a uniform probability over one true assumption and five false assumptions. Which is the sort of situation where subjective entropy seems quite appropriate.

Re: What Is Entropy?

#23

Post by fsckboy »

nihakue wrote: post #20

I'm not in any way qualified to have a take here, but I have one anyway: My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. So for example you could observe that a single coin flip is heads, a…

>purely a result of more likely things happening more often on average

according to your wording, no. if you have a perfect six sided die (or perfect two sided coin), none/neither of the outcomes are more likely at any point in time... yet something approximating entropy occurs after many repeated trials. what's expected to happen is the average thing even though it's never the most likely thing to happen.

you want to look at how repeated re-convolution of a function with itself always converges on the same gaussian function, no matter the shape of the starting function is (as long as it's not some pathological case, such as an impulse function... but even then, consider the convolution of the impulse function with the gaussian)

Re: What Is Entropy?

#24

Post by russdill »

nihakue wrote: post #20

I'm not in any way qualified to have a take here, but I have one anyway: My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. So for example you could observe that a single coin flip is heads, a…

This is based on entropy being closely tied to your knowledge of the system. It's one of many useful definitions of entropy.

Re: What Is Entropy?

#25

Post by karpathy »

What I never fully understood is that there is some implicit assumption about the dynamics of the system. So what that there are more microstates of some macrostate as far as counting is concerned? We also have to make assumptions about the dynamics, and in particular about some property that encourages mixing.

Re: What Is Entropy?

#26

Post by 867-5309 »

nihakue wrote: post #20

I'm not in any way qualified to have a take here, but I have one anyway: My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. So for example you could observe that a single coin flip is heads, a…

> 'actually be' and yet still 'appear to be'

esse quam videri

Re: What Is Entropy?

#27

Post by tomnicholas1 »

karpathy wrote: post #25

What I never fully understood is that there is some implicit assumption about the dynamics of the system. So what that there are more microstates of some macrostate as far as counting is concerned? We also have to make assumptions about the dynamics, and in particular about some property that encourages mixing.

Yes, that assumption is called the Ergodic Hypothesis, and generally justified in undergraduate statistical mechanics courses by proving and appealing to Liouville's theorem.

[1] https://en.wikipedia.org/wiki/Ergodic_hypothesis

Re: What Is Entropy?

#28

Post by TexanFeller »

I don’t see Sean Carroll’s musings mentioned yet, so repeating my previous comment:

Entropy got a lot more exciting to me after hearing Sean Carroll talk about it. He has a foundational/philosophical bent and likes to point out that there are competing definitions of entropy set on different philosophical foundations, one of them seemingly observer dependent: - https://youtu.be/x9COqqqsFtc?si=cQkfV5IpLC039Cl5 - https://youtu.be/XJ14ZO-e9NY?si=xi8idD5JmQbT5zxN

Leonard Susskind has lots of great talks and books about quantum information and calculating the entropy of black holes which led to a lot of wild new hypotheses.

Stephen Wolfram gave a long talk about the history of the concept of entropy which was pretty good: https://www.youtube.com/live/ocOHxPs1LQ0?si=zvQNsj_FEGbTX2R3

Re: What Is Entropy?

#29

Post by [deleted] »

karpathy wrote: post #25

What I never fully understood is that there is some implicit assumption about the dynamics of the system. So what that there are more microstates of some macrostate as far as counting is concerned? We also have to make assumptions about the dynamics, and in particular about some property that encourages mixing.

[deleted]

Re: What Is Entropy?

#30

Post by petsfed »

alganet wrote: post #3

Nowadays, it seems to be a buzzword to confuse people. We IT folk should find another word for disorder that increases over time, specially when that disorder has human factors (number of contributors, number of users, etc). It clearly cannot be treated in the same way as in chemistry.

When I use it in an IT (or honestly, any non-physics or non-physics) context, I typically mean "how many different ways can we do it with the same effective outcome?".

To whit, "contract entropy": how many different ways can a contractor technically fulfill the terms of the contract, and thus get paid? If your contract has high entropy, then there's a high probability that you'll pay your contractor to not actually achieve what you wanted.

Post reply on HN