Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Entropy of a Large Language Model output

nikkin.dev

Post reply on HN
Print view View original

21–30 of 67 posts

Re: Entropy of a Large Language Model output

#21

Post by TeMPOraL »

WhitneyLand wrote: post #2

> the output token of the LLM (black box) is not deterministic. Rather, it is a probability distribution over all the available tokens How is this not deterministic? Randomness is intentionally added via temperature.

There's extra randomness added accidentally in practice: inference is a massively parallelized set of matrix multiplications, and floating point math is not commutative - the randomness in execution order gets converted into a random FP error, so even setting temperature to 0 doesn't guarantee repeatable results.

Re: Entropy of a Large Language Model output

#23

Post by HeatrayEnjoyer »

TeMPOraL wrote: post #21
WhitneyLand wrote: post #2

> the output token of the LLM (black box) is not deterministic. Rather, it is a probability distribution over all the available tokens How is this not deterministic? Randomness is intentionally added via temperature.

There's extra randomness added accidentally in practice: inference is a massively parallelized set of matrix multiplications, and floating point math is not commutative - the randomness in execution order gets converted into a random FP error, so even setting temperature to 0 doesn't guarantee repeatable results.

Only if the inference software doesn't guarantee concurrency, which is CS 101

Re: Entropy of a Large Language Model output

#24

Post by HarHarVeryFunny »

jampekka wrote: post #20
HarHarVeryFunny wrote: post #5

Earlier quoted context omitted.

I think you're confusing training and inference. During training there are things like initialization, data shuffling and dropout that depend on random numbers. At inference time these don't apply.

Decoding (sampling) uses (pseudo) random numbers. Otherwise same prompt would always give the same response. Computing entropy generally does not. See e.g. https://huggingface.co/blog/how-to-generate

Sure - but that's not the output of the model itself, that's the process of (typically) randomly sampling from the output of the model.

Re: Entropy of a Large Language Model output

#25

Post by [deleted] »

cbzbc wrote: post #17
gwern wrote: post #10

You are observing "flattened logits" https://arxiv.org/pdf/2303.08774#page=12&org=openai . The entropy of Chat GPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens" or doing compression. A LLM like ChatGPT has been turned into an RL agent which seeks to maximize reward by taking the optimal action. It i…

Sorry, which particular part of that paper are you linking to, the graph at the top of that page doesn't seem to link to your comment?

[deleted]

Re: Entropy of a Large Language Model output

#26

Post by HarHarVeryFunny »

EncomLab wrote: post #13

We should stop using the term "black box" to mean "we don't know" when really it's "we could find out but it would be really hard". We can precisely determine the exact state of any digital system and track that state as it changes. In something as large as a LLM doing so is extremely complex, but complexity does not equal unknowable. These systems are still just software, with pre-defined operations executing in ord…

The usual use of the term "black box" is just that you are using/testing a system without knowing/assuming anything about what's inside. It doesn't imply that what's inside is complex or unknown - just unknown to an outside observer who can only see the box.

e.g.

In "black box" testing of a system you are just going to test based on the specifications of what the output/behavior should be for a given input. In contrast, in "white box" testing you leverage your knowledge of the internals of the box to test for things like edge cases that are apparent in the implementation, to test all code paths, etc.

Re: Entropy of a Large Language Model output

#27

Post by diggan »

Der_Einzige wrote: post #12
vletal wrote: post #8

Earlier quoted context omitted.

https://github.com/xjdr-alt/entropix

Entropix will get it's time in the sun, but for now, the LLM academic community is still 2 years behind the open source community. Min_p sampling is going to end up getting an oral about it at ICLR with the scores it's getting... https://openreview.net/forum?id=FBkpCyujtS

> the LLM academic community is still 2 years behind the open source community

Huh, isn't it the other way around? Thanks to the academic (and open) research about LLMs, we have any open source community around LLMs in the first place.

Re: Entropy of a Large Language Model output

#28

Post by EncomLab »

HarHarVeryFunny wrote: post #26
EncomLab wrote: post #13

We should stop using the term "black box" to mean "we don't know" when really it's "we could find out but it would be really hard". We can precisely determine the exact state of any digital system and track that state as it changes. In something as large as a LLM doing so is extremely complex, but complexity does not equal unknowable. These systems are still just software, with pre-defined operations executing in ord…

The usual use of the term "black box" is just that you are using/testing a system without knowing/assuming anything about what's inside. It doesn't imply that what's inside is complex or unknown - just unknown to an outside observer who can only see the box. e.g. In "black box" testing of a system you are just going to test based on the specifications of what the output/behavior should be for a given input. In contra…

Yes that is the definition - but that is not what is occurring her. We DO know exactly what is going on inside the system and can determine precisely from step to step the state of the entire system and the next state of the system. The author is making a claim based on woo that somehow this software operates differently than any other software at a fundamental level and that is not the case.

Re: Entropy of a Large Language Model output

#29

Post by EncomLab »

Ecoste wrote: post #16
EncomLab wrote: post #13

We should stop using the term "black box" to mean "we don't know" when really it's "we could find out but it would be really hard". We can precisely determine the exact state of any digital system and track that state as it changes. In something as large as a LLM doing so is extremely complex, but complexity does not equal unknowable. These systems are still just software, with pre-defined operations executing in ord…

So going by your definition what would be a true black box?

A starting point would be a system that does not require the use of a limited set of pre-defined operations to transform from one state to another state via the interpretation of a set of pre-existing instructions. This rules out any digital system entirely.

Re: Entropy of a Large Language Model output

#30

Post by HarHarVeryFunny »

gwern wrote: post #10

You are observing "flattened logits" https://arxiv.org/pdf/2303.08774#page=12&org=openai . The entropy of Chat GPT (as well as all other generative models which have been 'tuned' using RLHF, instruction-tuning, DPO, etc) is so low because it is not predicting "most likely tokens" or doing compression. A LLM like ChatGPT has been turned into an RL agent which seeks to maximize reward by taking the optimal action. It i…

Which is why models like o1 & o3, using heavy RL to boost reasoning performance, may perform worse in other areas where the greater diversity of output is needed.

Of course humans employ different thinking modes too - no harm in thinking like a stone cold programmer when you are programming, as long as you don't do it all the time.

Post reply on HN