Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Entropy of a Large Language Model output

nikkin.dev

Post reply on HN
Print view View original

61–67 of 67 posts

Re: Entropy of a Large Language Model output

#61

Post by nikkindev »

K0balt wrote: post #47

Low entropy is expected here, since the model is seeking a “best” answer based on reward training. But I see the same misconceptions as always around “hallucinations”. Incorrect output is just incorrect output. There is no difference in the function of the model, no malfunction. It is working exactly as it does for “correct “ answers. This is what makes the issue of incorrect output intractable. Some optimisation can…

Author here: Wholeheartedly agree with your comment on hallucination. I initially set out to answer the question “Will entropy help identify hallucination?” And soon realised that it doesn’t, for the same reasons you mentioned above. So I pivoted to just writing about the entropy measure in the post. And this is also reflected by how I started with hallucination and then quickly veered away from it. I’ll be more careful in future posts & conversations. Thanks!

Re: Entropy of a Large Language Model output

#62

Post by K0balt »

nikkindev wrote: post #61
K0balt wrote: post #47

Low entropy is expected here, since the model is seeking a “best” answer based on reward training. But I see the same misconceptions as always around “hallucinations”. Incorrect output is just incorrect output. There is no difference in the function of the model, no malfunction. It is working exactly as it does for “correct “ answers. This is what makes the issue of incorrect output intractable. Some optimisation can…

Author here: Wholeheartedly agree with your comment on hallucination. I initially set out to answer the question “Will entropy help identify hallucination?” And soon realised that it doesn’t, for the same reasons you mentioned above. So I pivoted to just writing about the entropy measure in the post. And this is also reflected by how I started with hallucination and then quickly veered away from it. I’ll be more care…

Nice post, really, and I think it will help some people to understand more about how LLMs work, especially helping fix the dogma about “LLMs just randomly select the next most likely word” which is kinda true but so many qualifiers and contextual details apply that the statement is more misleading than useful.

On undesired output, I would think it a great service to the field if we could come up with a better and earwormier word for “hallucinations” and somehow make it stick.

Right now we have half the literate world walking around thinking that LLMs are licking frogs, and it does nothing to help people understand how to think about model outputs or how to increase the utility of these fantastic culture / data mining tools in their own lives.

Re: Entropy of a Large Language Model output

#63

Post by apstroll »

wyager wrote: post #35
alew1 wrote: post #18

Earlier quoted context omitted.

"Temperature" doesn't make sense unless your model is predicting a distribution. You can't "temperature sample" a calculator, for instance. The output of the LLM is a predictive distribution over the next token; this is the formulation you will see in every paper on LLMs. It's true that you can do various things with that distribution other than sampling it: you can compute its entropy, you can find its mode (argmax)…

The temperature in LLMs is a parameter of a regularization step that determines how neuron activation levels get mapped to odds ratios. Zero temperature => fully deterministic The neuron activation levels do not inherently form or represent a probability distribution. That's something we've slapped on after the fact

Under a crossentropy loss the output activations do absolutely represent a probability distribution, since that is what we're modeling.

Re: Entropy of a Large Language Model output

#64

Post by paraschopra »

derefr wrote: post #42
gwern wrote: post #10

You are observing "flattened logits" https://arxiv.org/pdf/2303.08774#page=12&org=openai . The entropy of Chat GPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens" or doing compression. A LLM like ChatGPT has been turned into an RL agent which seeks to maximize reward by taking the optimal action. It i…

> The entropy of ChatGPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens" or doing compression. A LLM like ChatGPT has been turned into an RL agent which seeks to maximize reward by taking the optimal action. It is, ultimately, predicting what will manipulate the imaginary human rater into giving it a…

Interesting nuance. Goes on to suggest that these big models are multi-dimensional, complex monsters who we can only understand via low dimensional projections, and never as a whole.

Re: Entropy of a Large Language Model output

#65

Post by ramblenode »

derefr wrote: post #42
gwern wrote: post #10

You are observing "flattened logits" https://arxiv.org/pdf/2303.08774#page=12&org=openai . The entropy of Chat GPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens" or doing compression. A LLM like ChatGPT has been turned into an RL agent which seeks to maximize reward by taking the optimal action. It i…

> The entropy of ChatGPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens" or doing compression. A LLM like ChatGPT has been turned into an RL agent which seeks to maximize reward by taking the optimal action. It is, ultimately, predicting what will manipulate the imaginary human rater into giving it a…

I would like to see this turned into a blog post. Could even be a series.

Re: Entropy of a Large Language Model output

#67

Post by larodi »

BobbyJo wrote: post #58
larodi wrote: post #52

Earlier quoted context omitted.

> Reasoning scales all the way up to the discovery of quaternions and general relativity, often That would be true only if all that we grant for based/true/fact came through reasoning in a complete logical and awoke state. But it did not, and if you dig a little or more you'd find a lot of actual dreaming revelation, divine and all sorts of subconscious revelation that governs lives and also science.

I'd also like to point out serendipitous external input as well. Isaac Newton and watching the apple fall from the tree for instance. Often, thought processes are steered by external stimuli that happen to occur while the thought process is taking place.

its steered by magic or magick at best, and too many have tried to devise the principle behind it. math in all its beauty is too simple to grasp it yet if ever. noone does, that's the beauty of life. and nobody wants an ugly life.
Post reply on HN