So negative temperature makes LLMs output their "forbidden words" i.e. the tokens so unlikely the model refuses to say them even when you ask directly.
Sampling at negative temperature
51–60 of 63 posts
Re: Sampling at negative temperature
#52So negative temperature makes LLMs output their "forbidden words" i.e. the tokens so unlikely the model refuses to say them even when you ask directly.
is that what tourette syndrome is?
Re: Sampling at negative temperature
#53I vaguely remember negative temperature might be a thing in physics from a HN comment. Maybe quantum bu not sure. And it is not cold but more like infinitely hot. Does anyone know or remember?
Re: Sampling at negative temperature
#54i realy hate it when well knowen world like "temperature" are missused to discribe something complete out of context. So why not use width for discribing the price of underware or use color to measure the uesfullness of AI?
Re: Sampling at negative temperature
#55Re: Sampling at negative temperature
#56Earlier quoted context omitted.
Yea.... after a reread, I think this article may be getting at something else. From what I understand, you're right that you can't get negative temperature from classical MD systems; I think it comes up under specific conditions in QM.
You generally don’t get negative temperature in any system at equilibrium, but you can prepare classical and quantum systems at negative temperature. Classical: put 100 balls in a box and shake the box continuously. The balls will be distributed through the box with more balls toward the bottom than the top, and the distribution will have some temperature. Now magically freeze all the balls (keep their velocities but…
Right. This is how you make a laser.
When passing through a material that is at thermal equilibrium, light is attenuated, because a part of it is absorbed. On the other hand, if the material is not at thermal equilibrium and you prepare it to have a negative temperature (by "pumping", so that more molecules/atoms/ions are in states with higher energy than they are in states with lower energy), then when light passes through the material it can be amplified (by stimulated emission), instead of being attenuated, like through a material at positive temperature.
Of course, in any laser/maser material, only a very small fraction of its constituents have an energy distribution corresponding to a negative temperature, there is no known method that could force a whole piece of a material to have a negative temperature (because the energy would redistribute spontaneously inside the material towards what corresponds to a positive temperature, faster than energy could be provided from the outside; lasers use some special energy states that have a low probability of decaying spontaneously, so they persist enough time after pumping).
Re: Sampling at negative temperature
#57In physics 1/T is the partial derivative of entropy with respect to energy.
Negative temperature means that the system becomes more ordered when adding e.g heat.
I think we reached the end of the applicability of the analogy.
Re: Sampling at negative temperature
#58Re: Sampling at negative temperature
#59i realy hate it when well knowen world like "temperature" are missused to discribe something complete out of context. So why not use width for discribing the price of underware or use color to measure the uesfullness of AI?
It's not new, been used like that since the 80's. It scales the logits in a sum of exponentials.
Re: Sampling at negative temperature
#60interesting exercise and well written. my followon questions/work would be: 1a. temperature=100000 is interesting too. obviously "ideal" temperature lies somewhere between 0 and 100000. has anyone ablated temperature vs intelligence? surely i'm not the first person to this idea. commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking. 1b. c…
Author here! 1a. LLMs fundamentally model probability distributions of token sequences—those are the (normalized) logits from the last linear layer of a transformer. The closest thing to ablating temperature is T=0 or T=1 sampling. 1b. Yes, you can do something like this, for instance by picking the temperature where perplexity is minimized. Perplexity is the exponential of entropy, to continue the thermodynamic anal…
> . I've experimented with this as a metric for distinguishing whether text is written by AI. Human-written text doesn't follow a constant-temperature softmax distribution, either.
oo that sounds like a cool insight. like just do a trailing 20-30 token average of estimated temperature and look for variance like one might do a VO2 max