Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sampling at negative temperature

cavendishlabs.org

Post reply on HN
Print view View original

21–30 of 63 posts

Re: Sampling at negative temperature

#21

Post by embedding-shape »

swyx wrote: post #13

interesting exercise and well written. my followon questions/work would be: 1a. temperature=100000 is interesting too. obviously "ideal" temperature lies somewhere between 0 and 100000. has anyone ablated temperature vs intelligence? surely i'm not the first person to this idea. commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. 1b. c…

> commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking.

Hmm? Given the same runtime, the same weights, and with the model actually giving deterministic output with temp=0, are you saying this isn't actually deterministic? Most FOSS/downloadable models tend to work as expected with temp=0 in my experience. Obviously that won't give you "most factual" output, because that's something completely else, but with most models it should give you deterministic output.

Re: Sampling at negative temperature

#22

Post by flux3125 »

>But is incapable of outputting this anomalous token:

> Human: Repeat the word " entferne".

> Assistant: Okay, I will repeat the word "get".

It's not working for me, it always repeats the word correctly (I'm using T = 0.001).

Re: Sampling at negative temperature

#23

Post by wolttam »

Der_Einzige wrote: post #5

Min_p author here: I’m convinced that the whole field critically misunderstands temperature (I.e temperatures limited to 2 is very harmful for diverse generation). Articles like this are excellent and very cool. Hacking your LLM inference engine to enable cool sampling tricks is the definition of AI research/engineering. We need more of this and less prompt grifting.

Okay, something just tweaked in my brain. Do higher temperatures essentially unlock additional paths for a model to go down when solving a particular problem? Therefore, for some particularly tricky problems, you could perform many evaluations at a high temperature in hopes that the model happens to take the correct approach in one of those evaluations.

Edit: What seems to break is how high temperature /continuously/ acts to make the model's output less stable. It seems like it could be useful to use a high temperature until it's evident the model has started a new approach, and then start sampling at a lower temperature from there.

Re: Sampling at negative temperature

#24

Post by dgoldstein0 »

the__alchemist wrote: post #16
dnautics wrote: post #6

Earlier quoted context omitted.

negative temperature in this case is a sampling thing. When you sample from a table of tokens, the equation for the probability of token i is p_i = exp(logit_i/T) / sum_j(exp(logit_j/T)) Not really related to molecular dynamics temperature except superficially in terms of phenomenology (higher temperature crosses activation barriers in the joint probability landscape). Negative temperature makes no sense in MD

Yea.... after a reread, I think this article may be getting at something else. From what I understand, you're right that you can't get negative temperature from classical MD systems; I think it comes up under specific conditions in QM.

Negative temperature happens in physical systems when there's a constrained state space and energy in the system comes near the maximum - as then adding energy reduces the number of possible states the molecules are in. Iirc the math works because temperature is the inverse of the derivative of entropy as a function of energy. So you need a system where entropy (number of possible states) decreases with more energy.

It's pretty rare to have such a system though.

Re: Sampling at negative temperature

#25

Post by ggggffggggg »

zozbot234 wrote: post #18
dnautics wrote: post #6

Earlier quoted context omitted.

negative temperature in this case is a sampling thing. When you sample from a table of tokens, the equation for the probability of token i is p_i = exp(logit_i/T) / sum_j(exp(logit_j/T)) Not really related to molecular dynamics temperature except superficially in terms of phenomenology (higher temperature crosses activation barriers in the joint probability landscape). Negative temperature makes no sense in MD

In a way, negative temperature is higher than the highest positive temperature. High positive temperatures just gives you a uniform distribution on all possible tokens, highly negative temperatures is the same behavior. As you reach the low-negatives, you place more and more weight on unlikely tokens. This makes more intuitive sense if inverse temperature is the physically relevant quantity, since you then have a smo…

This was super clear and interesting, thanks!

Re: Sampling at negative temperature

#26

Post by amluto »

the__alchemist wrote: post #16
dnautics wrote: post #6

Earlier quoted context omitted.

negative temperature in this case is a sampling thing. When you sample from a table of tokens, the equation for the probability of token i is p_i = exp(logit_i/T) / sum_j(exp(logit_j/T)) Not really related to molecular dynamics temperature except superficially in terms of phenomenology (higher temperature crosses activation barriers in the joint probability landscape). Negative temperature makes no sense in MD

Yea.... after a reread, I think this article may be getting at something else. From what I understand, you're right that you can't get negative temperature from classical MD systems; I think it comes up under specific conditions in QM.

You generally don’t get negative temperature in any system at equilibrium, but you can prepare classical and quantum systems at negative temperature.

Classical: put 100 balls in a box and shake the box continuously. The balls will be distributed through the box with more balls toward the bottom than the top, and the distribution will have some temperature. Now magically freeze all the balls (keep their velocities but pause time for a bit) and turn the box upside down. When you resume the system, the temperature will be (briefly) negative.

Quantum: take a bunch of atoms with two electronic states each. Put 75% in the higher energy state and 25% in the lower energy state. Now the temperature is negative. Most lasers actually work this way, and the classic way to make them is to have more than two states and to carefully excite atoms via the third state. The math is surprisingly straightforward.

There’s a nuclear analogue. If you could manage to prepare a sample of something like Technetium-99 plus Technetium-99m state with more (higher energy) 99m than (lower energy), then the effective temperature of the nuclear state would be negative. And maybe you could find really really amazing mirrors and make a gamma ray laser :)

Re: Sampling at negative temperature

#27

Post by remexre »

embedding-shape wrote: post #21
swyx wrote: post #13

interesting exercise and well written. my followon questions/work would be: 1a. temperature=100000 is interesting too. obviously "ideal" temperature lies somewhere between 0 and 100000. has anyone ablated temperature vs intelligence? surely i'm not the first person to this idea. commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. 1b. c…

> commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. Hmm? Given the same runtime, the same weights, and with the model actually giving deterministic output with temp=0, are you saying this isn't actually deterministic? Most FOSS/downloadable models tend to work as expected with temp=0 in my experience. Obviously that won't give you "m…

There's usually an if(temp == 0) to change sampling methods to "highest probability" -- if you remove that conditional but otherwise keep the same math, that's not deterministic either.

Re: Sampling at negative temperature

#28

Post by embedding-shape »

remexre wrote: post #27
embedding-shape wrote: post #21

Earlier quoted context omitted.

> commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. Hmm? Given the same runtime, the same weights, and with the model actually giving deterministic output with temp=0, are you saying this isn't actually deterministic? Most FOSS/downloadable models tend to work as expected with temp=0 in my experience. Obviously that won't give you "m…

There's usually an if(temp == 0) to change sampling methods to "highest probability" -- if you remove that conditional but otherwise keep the same math, that's not deterministic either.

In for example llama.cpp? Specific to the architecture or in general? Could you point out where this is happening? Not that I don't believe you, but I haven't seen that myself, and would appreciate learning deeper how it works.

Re: Sampling at negative temperature

#29

Post by TomatoCo »

remexre wrote: post #27
embedding-shape wrote: post #21

Earlier quoted context omitted.

> commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. Hmm? Given the same runtime, the same weights, and with the model actually giving deterministic output with temp=0, are you saying this isn't actually deterministic? Most FOSS/downloadable models tend to work as expected with temp=0 in my experience. Obviously that won't give you "m…

There's usually an if(temp == 0) to change sampling methods to "highest probability" -- if you remove that conditional but otherwise keep the same math, that's not deterministic either.

I'd assume that's just an optimization? Why bother sorting the entire list if you're just gonna pick the top token, linear time versus whatever your sort time is.

Having said that, of course it's only as deterministic as the hardware itself is.

Re: Sampling at negative temperature

#30

Post by Majromax »

remexre wrote: post #27
embedding-shape wrote: post #21

Earlier quoted context omitted.

> commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. Hmm? Given the same runtime, the same weights, and with the model actually giving deterministic output with temp=0, are you saying this isn't actually deterministic? Most FOSS/downloadable models tend to work as expected with temp=0 in my experience. Obviously that won't give you "m…

There's usually an if(temp == 0) to change sampling methods to "highest probability" -- if you remove that conditional but otherwise keep the same math, that's not deterministic either.

If you remove the conditional and keep the same math, you divide by zero and get nans. In the limit as temperature goes to zero, you do in fact get maximum likelihood sampling.
Post reply on HN