Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Entropy of a Large Language Model output

nikkin.dev

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Entropy of a Large Language Model output

#51

Post by alew1 »

wyager wrote: post #35
alew1 wrote: post #18

Earlier quoted context omitted.

"Temperature" doesn't make sense unless your model is predicting a distribution. You can't "temperature sample" a calculator, for instance. The output of the LLM is a predictive distribution over the next token; this is the formulation you will see in every paper on LLMs. It's true that you can do various things with that distribution other than sampling it: you can compute its entropy, you can find its mode (argmax)…

The temperature in LLMs is a parameter of a regularization step that determines how neuron activation levels get mapped to odds ratios. Zero temperature => fully deterministic The neuron activation levels do not inherently form or represent a probability distribution. That's something we've slapped on after the fact

Any interpretation (including interpreting the inputs to the neural net as a "prompt") is "slapped on" in some sense—at some level, it's all just numbers being added, multiplied, and so on.

But I wouldn't call the probabilistic interpretation "after the fact." The entire training procedure that generated the LM weights (the pre-training as well as the RLHF post-training) is formulated based on the understanding that the LM predicts p(x_t | x_1, ..., x_{t-1}). For example, pretraining maximizes the log probability of the training data, and RLHF typically maximizes an objective that combines "expected reward [under the LLM's output probability distribution]" with "KL divergence between the pretraining distribution and the RLHF'd distribution" (a probabilistic quantity).

Re: Entropy of a Large Language Model output

#52

Post by larodi »

Vetch wrote: post #43
HarHarVeryFunny wrote: post #30

Earlier quoted context omitted.

Which is why models like o1 & o3, using heavy RL to boost reasoning performance, may perform worse in other areas where the greater diversity of output is needed. Of course humans employ different thinking modes too - no harm in thinking like a stone cold programmer when you are programming, as long as you don't do it all the time.

This seems wrong. Reasoning scales all the way up to the discovery of quaternions and general relativity, often requiring divergent thinking. Reasoning has a core aspect of maintaining uncertainty for better exploration and being able to tell when it's time to revisit the drawing board and start over from scratch. Being overconfident to the point of over-constraining possibility space will harm exploration, only work…

> Reasoning scales all the way up to the discovery of quaternions and general relativity, often

That would be true only if all that we grant for based/true/fact came through reasoning in a complete logical and awoke state. But it did not, and if you dig a little or more you'd find a lot of actual dreaming revelation, divine and all sorts of subconscious revelation that governs lives and also science.

Re: Entropy of a Large Language Model output

#53

Post by leptons »

gwern wrote: post #10

You are observing "flattened logits" https://arxiv.org/pdf/2303.08774#page=12&org=openai . The entropy of Chat GPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens" or doing compression. A LLM like ChatGPT has been turned into an RL agent which seeks to maximize reward by taking the optimal action. It i…

I wonder if at some point the LLMs will have consumed so much feedback, that when they are asked a question they will simply reply "42".

Re: Entropy of a Large Language Model output

#54

Post by Lerc »

There is an interesting aspect of this behaviour used in the byte latent transformer model.

Encoding tokens from source text can be done a number of ways, byte pair encoding, dictionaries etc.

You can also just encode text into tokens (or directly into embeddings) with yet another model.

The problem arises that if you are doing variable length tokens, how many characters do you put into any particular token, and then because that token must represent the text if you use it for decoding, where do you store count of characters stored in any particular token.

The byte latent transformer model solves this by using the entropy for the next character. A small character model receives the history character by character and predicts the next one. If the entropy spikes from low to high they count that as a token boundary. Decoding the same characters from the latent one at a time produces the same sequence and deterministically spikes at the same point in the decoding indicating that it is at the end of the token without the length being required to be explicitly encoded.

(disclaimer: My layman's view of it anyway, I may be completely wrong)

Re: Entropy of a Large Language Model output

#55

Post by throwaway314155 »

HarHarVeryFunny wrote: post #24
jampekka wrote: post #20

Earlier quoted context omitted.

Decoding (sampling) uses (pseudo) random numbers. Otherwise same prompt would always give the same response. Computing entropy generally does not. See e.g. https://huggingface.co/blog/how-to-generate

Sure - but that's not the output of the model itself, that's the process of (typically) randomly sampling from the output of the model.

Right, sampling from a model, also known as *inference* (for LLM's).

The inference here is perhaps less pure than what you refer to but you're talking to human beings; there's no need for heavy pedantry.

Re: Entropy of a Large Language Model output

#56

Post by kbelder »

observationist wrote: post #37
EncomLab wrote: post #13

We should stop using the term "black box" to mean "we don't know" when really it's "we could find out but it would be really hard". We can precisely determine the exact state of any digital system and track that state as it changes. In something as large as a LLM doing so is extremely complex, but complexity does not equal unknowable. These systems are still just software, with pre-defined operations executing in ord…

Something like: A black box is unknowable, a gray box can be figured out in principle, a white box is fully known. A pocket calculator is fully known. LLMs are (dark) gray boxes - we can, in principle, figure out any particular sequence of computations, at any particular level you want to look at, but doing so is extremely tedious. Tools are being researched and developed to make this better, and mechinterp makes pro…

One important point (I think) is whether the cause or outcome of the box can be understood or predicted without full emulation of the entire box. Can it be distilled down to a more simple set of rules, or is it a chaotic system that turns into a different system if any part of it is removed?

That is, can you trace unequivocally the reason an LLM produced a certain token without, in effect, recreating the LLM and asking it the same question again?

Re: Entropy of a Large Language Model output

#57

Post by hansvm »

fancyfredbot wrote: post #34
HarHarVeryFunny wrote: post #11

Earlier quoted context omitted.

The author of the article seems confused, saying: "The important thing to remember is that the output token of the LLM (black box) is not deterministic. Rather, it is a probability distribution over all the available tokens in the vocabulary." He is saying that there is non-determinism in the output of the LLM (i.e. in these probability distributions), when in fact the randomness only comes from choosing to use a ran…

The author is saying that the output token is not deterministic. I don't think they said the distribution was stochastic. Even so the distribution of the second token output by the model would be stochastic (unless you condition on the first token). So in that sense there may also be a stochastic probability distribution.

Mostly unrelated (I agree with you, and I'm some ancestory comment you're responding to with the same line of thinking), I have built a couple LLMs where the distribution itself is stochastic. That's not key to how they work as a black box, but much like how quicksort has certain performance characteristics I did find it advantageous to introduce randomness into the model itself.

You could still easily model the next token as a conditional probability distribution though if you wanted; the computation of entropy just might be a bit spendier.

Re: Entropy of a Large Language Model output

#58

Post by BobbyJo »

larodi wrote: post #52
Vetch wrote: post #43

Earlier quoted context omitted.

This seems wrong. Reasoning scales all the way up to the discovery of quaternions and general relativity, often requiring divergent thinking. Reasoning has a core aspect of maintaining uncertainty for better exploration and being able to tell when it's time to revisit the drawing board and start over from scratch. Being overconfident to the point of over-constraining possibility space will harm exploration, only work…

> Reasoning scales all the way up to the discovery of quaternions and general relativity, often That would be true only if all that we grant for based/true/fact came through reasoning in a complete logical and awoke state. But it did not, and if you dig a little or more you'd find a lot of actual dreaming revelation, divine and all sorts of subconscious revelation that governs lives and also science.

I'd also like to point out serendipitous external input as well. Isaac Newton and watching the apple fall from the tree for instance. Often, thought processes are steered by external stimuli that happen to occur while the thought process is taking place.

Re: Entropy of a Large Language Model output

#59

Post by pizza »

HeatrayEnjoyer wrote: post #23
TeMPOraL wrote: post #21

Earlier quoted context omitted.

There's extra randomness added accidentally in practice: inference is a massively parallelized set of matrix multiplications, and floating point math is not commutative - the randomness in execution order gets converted into a random FP error, so even setting temperature to 0 doesn't guarantee repeatable results.

Only if the inference software doesn't guarantee concurrency, which is CS 101

This sort of nondeterministic scheduling of non associative floating point ops is essentially running at the level of GPU firmware, so, I would imagine that in this case, Nvidia is aware.

Re: Entropy of a Large Language Model output

#60

Post by pizza »

netruk44 wrote: post #4

I wonder if we could combine ‘thinking’ models (which write thoughts out before replying) with a mechanism they can use to check their own entropy as they’re writing output. Maybe it could eventually learn when it needs to have a low entropy token (to produce a more-likely-to-be-factual statement) and then we can finally have models that actually definitely know when to say “Sorry, I don’t seem to have a good answer…

There's a paper that probed how strongly a model would focus on prompt-supplied tokens when generating a response as a signal that it was trying to use the prompt as the source of information as opposed to knowledge it had been trained on. Ie, how much it was trying to lie based on it assuming that the information in the prompt was true, as opposed to having a rich internal model of the thing that is being verified. It looks like it works, sort of, sometimes, when you have access to the actual labels. The results from this work, in the more real-world unsupervised setting, are better than random, sure, but not good enough to really be exciting or reliable.

https://arxiv.org/html/2402.03563v1

Post reply on HN