Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sampling at negative temperature

cavendishlabs.org

Post reply on HN
Print view View original

41–50 of 63 posts

Re: Sampling at negative temperature

#41

Post by dnautics »

TomatoCo wrote: post #29
remexre wrote: post #27

Earlier quoted context omitted.

There's usually an if(temp == 0) to change sampling methods to "highest probability" -- if you remove that conditional but otherwise keep the same math, that's not deterministic either.

I'd assume that's just an optimization? Why bother sorting the entire list if you're just gonna pick the top token, linear time versus whatever your sort time is. Having said that, of course it's only as deterministic as the hardware itself is.

The likelihood that top-two is close enough to be hardware dependent is pretty low. IIUC It's more of an issue when you are using other picking methods.

Re: Sampling at negative temperature

#42

Post by vlovich123 »

swyx wrote: post #13

interesting exercise and well written. my followon questions/work would be: 1a. temperature=100000 is interesting too. obviously "ideal" temperature lies somewhere between 0 and 100000. has anyone ablated temperature vs intelligence? surely i'm not the first person to this idea. commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. 1b. c…

Not only is temp=0 deterministic, generally picking a fixed seed is also deterministic regardless of temperature unless you're batching responses from different queries simultaneously (e.g. OpenAI).

Re: Sampling at negative temperature

#43

Post by layer8 »

the__alchemist wrote: post #3

This is so cool! I just learned about this last week. For reference, I do molecular dynamics (my own engine, in rust), and measuring temperature is an important part of the simulation. (So you can nudge it to a target temperature, for example). An important component of this calculation is the degrees of freedom of the system. Calculating this depends on your model. For example, are you representing atoms that can ea…

> This is so cool!

Negative temperature tends to be that. ;)

Re: Sampling at negative temperature

#44

Post by turzmo »

layer8 wrote: post #43
the__alchemist wrote: post #3

This is so cool! I just learned about this last week. For reference, I do molecular dynamics (my own engine, in rust), and measuring temperature is an important part of the simulation. (So you can nudge it to a target temperature, for example). An important component of this calculation is the degrees of freedom of the system. Calculating this depends on your model. For example, are you representing atoms that can ea…

> This is so cool! Negative temperature tends to be that. ;)

Negative temperatures are hotter than positive ones, in fact

Re: Sampling at negative temperature

#45

Post by programjames »

bjourne wrote: post #14
Der_Einzige wrote: post #5

Min_p author here: I’m convinced that the whole field critically misunderstands temperature (I.e temperatures limited to 2 is very harmful for diverse generation). Articles like this are excellent and very cool. Hacking your LLM inference engine to enable cool sampling tricks is the definition of AI research/engineering. We need more of this and less prompt grifting.

Correct me if I'm wrong, but the problem is that it is almost impossible to evaluate sampling methods. You can't just look at perplexity and conclude that A is better than B. So you need large-scale expensive human evaluations. Even if you have those it is difficult to extrapolate results since what sampling method works best depends on the dataset(s).

I think you can try maximizing the free energy E[reward] + temperature*entropy?

Re: Sampling at negative temperature

#47

Post by GRiMe2D »

wolttam wrote: post #23
Der_Einzige wrote: post #5

Min_p author here: I’m convinced that the whole field critically misunderstands temperature (I.e temperatures limited to 2 is very harmful for diverse generation). Articles like this are excellent and very cool. Hacking your LLM inference engine to enable cool sampling tricks is the definition of AI research/engineering. We need more of this and less prompt grifting.

Okay, something just tweaked in my brain. Do higher temperatures essentially unlock additional paths for a model to go down when solving a particular problem? Therefore, for some particularly tricky problems, you could perform many evaluations at a high temperature in hopes that the model happens to take the correct approach in one of those evaluations. Edit: What seems to break is how high temperature /continuously/…

I think yes. Recently I was experimenting with NEAT and HyperNEAT solutions and found this site. At the bottom it explains how novelty yields far more optimal solutions. I would assume that reasonably high temperature may also result more interesting solutions from LLM

https://blog.lunatech.com/posts/2024-02-29-the-neat-algorith...

Re: Sampling at negative temperature

#48

Post by eru »

dnautics wrote: post #40
Majromax wrote: post #30

Earlier quoted context omitted.

If you remove the conditional and keep the same math, you divide by zero and get nans. In the limit as temperature goes to zero, you do in fact get maximum likelihood sampling.

if (t==0) argmax(logits) else pick(logits)

You are ignoring the limit taking.

Re: Sampling at negative temperature

#49

Post by omneity »

swyx wrote: post #33
embedding-shape wrote: post #21

Earlier quoted context omitted.

> commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. Hmm? Given the same runtime, the same weights, and with the model actually giving deterministic output with temp=0, are you saying this isn't actually deterministic? Most FOSS/downloadable models tend to work as expected with temp=0 in my experience. Obviously that won't give you "m…

"What might be more surprising is that even when we adjust the temperature down to 0This means that the LLM always chooses the highest probability token, which is called greedy sampling. (thus making the sampling theoretically deterministic), LLM APIs are still not deterministic in practice (see past discussions here, here, or here)" https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

That's just an implementation artifact and not a fundamental fact of life.

https://docs.vllm.ai/en/latest/features/batch_invariance/

Re: Sampling at negative temperature

#50

Post by bjourne »

programjames wrote: post #45
bjourne wrote: post #14

Earlier quoted context omitted.

Correct me if I'm wrong, but the problem is that it is almost impossible to evaluate sampling methods. You can't just look at perplexity and conclude that A is better than B. So you need large-scale expensive human evaluations. Even if you have those it is difficult to extrapolate results since what sampling method works best depends on the dataset(s).

I think you can try maximizing the free energy E[reward] + temperature*entropy?

How do you know that generates high quality text?
Post reply on HN