Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Entropy of a Large Language Model output

nikkin.dev

Post reply on HN
Print view View original

11–20 of 67 posts

Re: Entropy of a Large Language Model output

#11

Post by HarHarVeryFunny »

hansvm wrote: post #6
WhitneyLand wrote: post #2

> the output token of the LLM (black box) is not deterministic. Rather, it is a probability distribution over all the available tokens How is this not deterministic? Randomness is intentionally added via temperature.

The output "token" Yes, you can sample deterministically, but that's some combination of computationally intractable and only useful on a small subset of problems. The black box outputting a non-deterministic token is a close enough approximation for most people.

The author of the article seems confused, saying:

"The important thing to remember is that the output token of the LLM (black box) is not deterministic. Rather, it is a probability distribution over all the available tokens in the vocabulary."

He is saying that there is non-determinism in the output of the LLM (i.e. in these probability distributions), when in fact the randomness only comes from choosing to use a random number generator to sample from this output.

Re: Entropy of a Large Language Model output

#12

Post by Der_Einzige »

vletal wrote: post #8
netruk44 wrote: post #4

I wonder if we could combine ‘thinking’ models (which write thoughts out before replying) with a mechanism they can use to check their own entropy as they’re writing output. Maybe it could eventually learn when it needs to have a low entropy token (to produce a more-likely-to-be-factual statement) and then we can finally have models that actually definitely know when to say “Sorry, I don’t seem to have a good answer…

https://github.com/xjdr-alt/entropix

Entropix will get it's time in the sun, but for now, the LLM academic community is still 2 years behind the open source community. Min_p sampling is going to end up getting an oral about it at ICLR with the scores it's getting...

https://openreview.net/forum?id=FBkpCyujtS

Re: Entropy of a Large Language Model output

#13

Post by EncomLab »

We should stop using the term "black box" to mean "we don't know" when really it's "we could find out but it would be really hard".

We can precisely determine the exact state of any digital system and track that state as it changes. In something as large as a LLM doing so is extremely complex, but complexity does not equal unknowable.

These systems are still just software, with pre-defined operations executing in order like any other piece of software. A CPU does not enter some mysterious woo "LLM black box" state that is somehow fundamentally different than running any other software, and it's these imprecise terms that lead to so much of the hype.

Re: Entropy of a Large Language Model output

#14

Post by Der_Einzige »

apstroll wrote: post #7
WhitneyLand wrote: post #2

> the output token of the LLM (black box) is not deterministic. Rather, it is a probability distribution over all the available tokens How is this not deterministic? Randomness is intentionally added via temperature.

The output distribution is deterministic, the output token is sampled from the output distribution, and is therefore not deterministic. Temperature modulates the output distribution, but sitting it to 0 (i.e. argmax sampling) is not the norm.

Running temperature of zero/greedy sampling (what you call "argmax sampling") is EXTREMELY common.

LLMs are basically "deterministic" when using greedy sampling except for either MoE related shenanigans (what historically prevented determinism in ChatGPT) or due to floating point related issues (GPU related). In practice, LLMs are in fact basically "deterministic" except for the sampling/temperature stuff that we add at the very end.

Re: Entropy of a Large Language Model output

#15

Post by saurik »

EncomLab wrote: post #13

We should stop using the term "black box" to mean "we don't know" when really it's "we could find out but it would be really hard". We can precisely determine the exact state of any digital system and track that state as it changes. In something as large as a LLM doing so is extremely complex, but complexity does not equal unknowable. These systems are still just software, with pre-defined operations executing in ord…

This is much more similar to the technique of obfuscating encryption algorithms for DRM schemes that I believe is often called "white-box cryptography".

Re: Entropy of a Large Language Model output

#16

Post by Ecoste »

EncomLab wrote: post #13

We should stop using the term "black box" to mean "we don't know" when really it's "we could find out but it would be really hard". We can precisely determine the exact state of any digital system and track that state as it changes. In something as large as a LLM doing so is extremely complex, but complexity does not equal unknowable. These systems are still just software, with pre-defined operations executing in ord…

So going by your definition what would be a true black box?

Re: Entropy of a Large Language Model output

#17

Post by cbzbc »

gwern wrote: post #10

You are observing "flattened logits" https://arxiv.org/pdf/2303.08774#page=12&org=openai . The entropy of Chat GPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens" or doing compression. A LLM like ChatGPT has been turned into an RL agent which seeks to maximize reward by taking the optimal action. It i…

Sorry, which particular part of that paper are you linking to, the graph at the top of that page doesn't seem to link to your comment?

Re: Entropy of a Large Language Model output

#18

Post by alew1 »

WhitneyLand wrote: post #2

> the output token of the LLM (black box) is not deterministic. Rather, it is a probability distribution over all the available tokens How is this not deterministic? Randomness is intentionally added via temperature.

"Temperature" doesn't make sense unless your model is predicting a distribution. You can't "temperature sample" a calculator, for instance. The output of the LLM is a predictive distribution over the next token; this is the formulation you will see in every paper on LLMs. It's true that you can do various things with that distribution other than sampling it: you can compute its entropy, you can find its mode (argmax), etc., but the type signature of the LLM itself is `prompt -> probability distribution over next tokens`.

Re: Entropy of a Large Language Model output

#20

Post by jampekka »

HarHarVeryFunny wrote: post #5
cjtrowbridge wrote: post #3

Earlier quoted context omitted.

Entropy is also added via a random seed. The model is only deterministic if you use the same random seed.

I think you're confusing training and inference. During training there are things like initialization, data shuffling and dropout that depend on random numbers. At inference time these don't apply.

Decoding (sampling) uses (pseudo) random numbers. Otherwise same prompt would always give the same response.

Computing entropy generally does not.

See e.g. https://huggingface.co/blog/how-to-generate

Post reply on HN