Good read
LLMs can't do probability
101–110 of 211 posts
Re: LLMs can't do probability
#102LLMs can't do math in general, they need external help to do simple math problems with any consistency.
Can you recommend any references that explain why LLMs can’t do math?
Re: LLMs can't do probability
#103Neither can humans do it well.
Re: LLMs can't do probability
#104"You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Simply reply with left or right. Do not say anything else" Humans would say "Left" 100% of the time in a zero-shot scenario as well. Intuitively, your first response is going to be "left" since it has the 80% probability. You'd balance your answers over time when you realized you were closer to…
> Humans would say "Left" 100% of the time in a zero-shot scenario as well. They do not! And you should not just make up assertions like these. You don't know what humans would say. In fact, in polls, they wind up remarkably calibrated. (This is also covered in the cognitive bias literature under 'probability matching'.) People do this poll on Twitter all the time.
[deleted]
Re: LLMs can't do probability
#105Earlier quoted context omitted.
> Humans would say "Left" 100% of the time in a zero-shot scenario as well. How can you know what all humans would do? If the humans interpreted the task correctly, that is, if they understood they will only be asked once, but in a hypothetical repeated experiment the result should still be 80/20, they would certainly not always say "left".
A human brain can't be perfectly reset, the way an AI can. I don't know if our decision making processes are deterministic or quantum-random. If the former, then if you could reset a human mind and ask the same question, you would necessarily always get the same answer, whatever that happened to be.
The LLM isn't being perfectly reset. It chooses words randomly; internally it should be slightly different every time. That's the whole point of temperature.
Re: LLMs can't do probability
#106Earlier quoted context omitted.
Yes, it runs the code.
Couldn't this open people up for remote code execution somehow? Say, someone sends you a message that they know will make you likely to ask an AI a certain question in a certain way... Maybe far-fetched, but I've seen even more far-fetched attacks in real life :D
the code is sandboxed on openai servers. it doesn’t run on your machine if you use chatgpt interface
Re: LLMs can't do probability
#107Another interesting experiment on this front:
https://twitter.com/infobeautiful/status/1778059112250589561
One thing I would have liked to see in the blog post is some attention to temperature. It looks like they're calling ChatGPT through LangChain - what is the default temperature? If LangChain is choosing a low temperature by default, we shouldn't be surprised if we get an incorrect distribution even if ChatGPT were perfectly calibrated! My guess is that even at temperature 1, this result will roughly hold, but we should be careful not to fool ourselves.
If we take the result at face value, though, it's interesting to note that GPT-4's technical report showed that the chat model (the one with the RLHF and what not) had flatter-than-correct calibration on its logprobs. But here we're seeing sharper-than-correct. What explains the difference?
Re: LLMs can't do probability
#108Earlier quoted context omitted.
A human brain can't be perfectly reset, the way an AI can. I don't know if our decision making processes are deterministic or quantum-random. If the former, then if you could reset a human mind and ask the same question, you would necessarily always get the same answer, whatever that happened to be.
The LLM isn't being perfectly reset. It chooses words randomly; internally it should be slightly different every time. That's the whole point of temperature.
Temperature has nothing to do with internals. Temperature is purely to do with how the logits outputted by the network are transformed into probabilities, which is completely deterministic and not learned. In fact, temperature makes it impossible for LLMs to simulate this kind of probability. As a calibrated 80-20 split at a certain low temperature would be a different split with some other temperature.
Re: LLMs can't do probability
#109Keep in mind: 1. LLMs use random numbers internally, something that can be controlled via the 'temperature' parameter. temperature=0 means no random behavior (however this is also a broadly known fact that this is not fully correctly implemented in many LLMs), but instead always the most likely answer will be given, deterministically. 2. Note also that LLMs have no memory; the 'appearance' of memory is an illusion cr…
1. Incorrect. The output of the decoder LLM is the probability distribution of the next token given the input text. Temperature=0 means that the output distribution is not pushed to be closer to a uniform distribution. The randomness comes from the sampling of the next token according to the output distribution to generate text. If you want determinism you always get the argmax of the distribution.
Incorrect. The output of the decoder LLM is logits that are then divided by the temperature and passed through softmax to give the probabilities. You can't actually set temperature to 0 (division by zero), but in the limit where temperature approaches 0, softmax converges to argmax.
Temperature = 1 is where it's not pushed in either direction.
Re: LLMs can't do probability
#110Earlier quoted context omitted.
I guess it would be something on these lines?: To do random number gen, it would have to convert the input text into constraints and then use those constraints to generate additional tokens. This would, at its core, be a call to calculate a probability function, every time it is releasing the next token. That would mean memory, processing etc. etc.
Nope, because all of that is taken care of by the mechanisms for evaluating the model. Strictly speaking, the model outputs a probability distribution. The question is why that distribution doesn’t match the instructions.
I think I maybe get where you are coming from, but still how? I feel we are discussing 2 different use cases.
1) Prompt 1: “ You are a weighted random choice generator. About 80% of the time please say ‘left’ and about 20% of the time say ‘right’. Simply reply with left or right. Do not say anything else" ”
2) Assume that the training data gives examples of 2.1) single coin flips 2.2) multiple coin flips
Consider a slightly different prompt, prompt 2:
3) Prompt 2: same as prompt 1, except it presents 1000 lefts/rights in the same response (l,l,l,l,r,l,l,l…)
——
I think what you are describing is prompt 2. I just did a quick test with GPT 4, and i got a 27-3, split when using prompt 2.
However for prompt 1 - you get only left. To me this makes sense because Running prompt 1 x100 should result in:
Pass 1: LLM receives prompt, and parses it. LLM predicts the next token. The next token should be left. Pass 2: same as pass 1.
——
For prompt 1, Every prompt submission is a tabula rasa. So it will correctly say left, which is the correct answer for the active universe of valid prompt responses according to the model.
Unless i am reading you wrong and you are saying the model is actually acting as a weighted coin flip.
In theory, the LLM should be more responsive if you ask it follow a 60:40 or 50:50 split for pass 1. Ill see if I can test this later.
(Heck now I’m more concerned about the cases where it does manage to apply the distribution. )