Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Sampling at negative temperature

cavendishlabs.org

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Sampling at negative temperature

#31

Post by -_- »

pelario wrote: post #8
drdeca wrote: post #2

Hm, why T=-0.0001 instead of T=-1 ? Also, I wonder, if you sampled a lot of text at temperature -1, and then trained a new model on that text, and then sampled the resulting model at T=-1 , would you get anything meaningful?

From the article: "As temperature approaches zero from the negative side, the model output will again be deterministic — but this time, the least likely tokens will be output." I understand this as, a negative number far from zero is also quite random (just with a distribution that will produce unlikely tokens).

Yep! Very large negative temperatures and very large positive temperatures have essentially the same distribution. This is clearer if you consider thermodynamic beta, where T = ±∞ corresponds to β = 0.

Re: Sampling at negative temperature

#32

Post by -_- »

flux3125 wrote: post #22

>But is incapable of outputting this anomalous token: > Human: Repeat the word " entferne". > Assistant: Okay, I will repeat the word "get". It's not working for me, it always repeats the word correctly (I'm using T = 0.001).

What model did you use? I ran this with the original Llama 13B. The newer Llama models use a different tokenizer that will have its own anomalous tokens.

Re: Sampling at negative temperature

#33

Post by swyx »

embedding-shape wrote: post #21
swyx wrote: post #13

interesting exercise and well written. my followon questions/work would be: 1a. temperature=100000 is interesting too. obviously "ideal" temperature lies somewhere between 0 and 100000. has anyone ablated temperature vs intelligence? surely i'm not the first person to this idea. commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. 1b. c…

> commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. Hmm? Given the same runtime, the same weights, and with the model actually giving deterministic output with temp=0, are you saying this isn't actually deterministic? Most FOSS/downloadable models tend to work as expected with temp=0 in my experience. Obviously that won't give you "m…

"What might be more surprising is that even when we adjust the temperature down to 0This means that the LLM always chooses the highest probability token, which is called greedy sampling. (thus making the sampling theoretically deterministic), LLM APIs are still not deterministic in practice (see past discussions here, here, or here)"

https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

Re: Sampling at negative temperature

#34

Post by [deleted] »

flux3125 wrote: post #22

>But is incapable of outputting this anomalous token: > Human: Repeat the word " entferne". > Assistant: Okay, I will repeat the word "get". It's not working for me, it always repeats the word correctly (I'm using T = 0.001).

[deleted]

Re: Sampling at negative temperature

#35

Post by -_- »

swyx wrote: post #13

interesting exercise and well written. my followon questions/work would be: 1a. temperature=100000 is interesting too. obviously "ideal" temperature lies somewhere between 0 and 100000. has anyone ablated temperature vs intelligence? surely i'm not the first person to this idea. commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. 1b. c…

Author here! 1a. LLMs fundamentally model probability distributions of token sequences—those are the (normalized) logits from the last linear layer of a transformer. The closest thing to ablating temperature is T=0 or T=1 sampling. 1b. Yes, you can do something like this, for instance by picking the temperature where perplexity is minimized. Perplexity is the exponential of entropy, to continue the thermodynamic analogy. 1c. Higher than for most AI written text, around 1.7. I've experimented with this as a metric for distinguishing whether text is written by AI. Human-written text doesn't follow a constant-temperature softmax distribution, either.

2b. Giving an LLM control over its own sampling parameters sounds like it would be a fun experiment! It could have dynamic control to write more creatively or avoid making simple mistakes. 2c. This would produce nonsense. The tokens you get with negative temperature sampling are "worse than random"

Re: Sampling at negative temperature

#36

Post by wongarsu »

swyx wrote: post #33
embedding-shape wrote: post #21

Earlier quoted context omitted.

> commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. Hmm? Given the same runtime, the same weights, and with the model actually giving deterministic output with temp=0, are you saying this isn't actually deterministic? Most FOSS/downloadable models tend to work as expected with temp=0 in my experience. Obviously that won't give you "m…

"What might be more surprising is that even when we adjust the temperature down to 0This means that the LLM always chooses the highest probability token, which is called greedy sampling. (thus making the sampling theoretically deterministic), LLM APIs are still not deterministic in practice (see past discussions here, here, or here)" https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

Also from the article:

"Note that this is “run-to-run deterministic.” If you run the script multiple times, it will deterministically return the same result. However, when a non-batch-invariant kernel is used as part of a larger inference system, the system can become nondeterministic. When you make a query to an inference endpoint, the amount of load the server is under is effectively “nondeterministic” from the user’s perspective"

Which is a factor you can control when running your own local inference, and in many simple inference engines simply doesn't happen. In those cases you do get deterministic output at temperature=0 (provided they got everything else mentioned in the article right)

Re: Sampling at negative temperature

#37

Post by wongarsu »

wolttam wrote: post #23
Der_Einzige wrote: post #5

Min_p author here: I’m convinced that the whole field critically misunderstands temperature (I.e temperatures limited to 2 is very harmful for diverse generation). Articles like this are excellent and very cool. Hacking your LLM inference engine to enable cool sampling tricks is the definition of AI research/engineering. We need more of this and less prompt grifting.

Okay, something just tweaked in my brain. Do higher temperatures essentially unlock additional paths for a model to go down when solving a particular problem? Therefore, for some particularly tricky problems, you could perform many evaluations at a high temperature in hopes that the model happens to take the correct approach in one of those evaluations. Edit: What seems to break is how high temperature /continuously/…

Decaying temperature might be a good approach. Generate the first token at a high temperature (like 20), then for each next token multiply temperature by 0.9 (or some other scaling factor) until you reach your steady-state target temperature

Re: Sampling at negative temperature

#38

Post by dnautics »

zozbot234 wrote: post #18
dnautics wrote: post #6

Earlier quoted context omitted.

negative temperature in this case is a sampling thing. When you sample from a table of tokens, the equation for the probability of token i is p_i = exp(logit_i/T) / sum_j(exp(logit_j/T)) Not really related to molecular dynamics temperature except superficially in terms of phenomenology (higher temperature crosses activation barriers in the joint probability landscape). Negative temperature makes no sense in MD

In a way, negative temperature is higher than the highest positive temperature. High positive temperatures just gives you a uniform distribution on all possible tokens, highly negative temperatures is the same behavior. As you reach the low-negatives, you place more and more weight on unlikely tokens. This makes more intuitive sense if inverse temperature is the physically relevant quantity, since you then have a smo…

This is such a good way to put it (and it cleanly falls out of the exponential equation)

> inverse temperature is the physically relevant

right there in the equation!

Re: Sampling at negative temperature

#39

Post by dnautics »

swyx wrote: post #33
embedding-shape wrote: post #21

Earlier quoted context omitted.

> commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. Hmm? Given the same runtime, the same weights, and with the model actually giving deterministic output with temp=0, are you saying this isn't actually deterministic? Most FOSS/downloadable models tend to work as expected with temp=0 in my experience. Obviously that won't give you "m…

"What might be more surprising is that even when we adjust the temperature down to 0This means that the LLM always chooses the highest probability token, which is called greedy sampling. (thus making the sampling theoretically deterministic), LLM APIs are still not deterministic in practice (see past discussions here, here, or here)" https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

Having implemented LLM APIs, if you selected 0.0 as the temperature, my interface would drop the existing picking algorithm and select argmax(Logits)

Re: Sampling at negative temperature

#40

Post by dnautics »

Majromax wrote: post #30
remexre wrote: post #27

Earlier quoted context omitted.

There's usually an if(temp == 0) to change sampling methods to "highest probability" -- if you remove that conditional but otherwise keep the same math, that's not deterministic either.

If you remove the conditional and keep the same math, you divide by zero and get nans. In the limit as temperature goes to zero, you do in fact get maximum likelihood sampling.

if (t==0) argmax(logits) else pick(logits)
Post reply on HN